Ontology-augmented interface

ABSTRACT

Provided is a process including obtaining a set of natural-language text documents that discuss a topic, the set of documents containing different states of knowledge about the topic at different times. The process includes selecting an ontology from among a plurality of ontologies that correspond to different domains of knowledge, the selection being based on the ontology corresponding to a domain of knowledge including the topic. The process includes identifying concepts discussed in the documents using the ontology and detecting changes in at least some of the concepts over time based on differences between discussion of the concepts in documents authored at different times. The process includes updating natural language instructions on the topic based on the detected changes in the concepts and storing the updated natural language instructions in memory.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent claims the benefit of U.S. Provisional Patent Application62/993,122, filed 23 Mar. 2020, titled “MULTI-SCALE SUPPORT FOR NATURALLANGUAGE UNDERSTANDING.” The entirety of the content of eachaforementioned patent filing is hereby incorporated by reference.

BACKGROUND 1. Field

The present disclosure relates generally to machine learning and, morespecifically, to natural language processing for cross-context naturallanguage model generation.

2. Description of the Related Art

Natural language understanding (NLU) is a sub-field of natural languageprocessing (NLP). NLU operations and NLP operations are expected toimpact a broad spectrum of disciplines such as computer operations,medicine, education, and finance. NLU operations can be used whenstoring, retrieving, or analyzing information in such fields.Furthermore, NLU operations can be performed on server-side devices orclient-side devices and can provide information in response to queries.

SUMMARY

The following is a non-exhaustive listing of some aspects of the presenttechniques. These and other aspects are described in the followingdisclosure.

Some aspects include a process that includes obtaining, via a network, afirst message from a client computing device, where the first messageidentifies a natural-language text document. The process includessending a user interface (UI) to the client computing device via aresponse to the first message. The UI includes: a display of thenatural-language text document that includes n-grams associated with aset of ontologies, a visual indicator indicating an ontology-linkedn-gram of the natural-language text document is mapped to an entry of anontology, and a UI element that, when interacted with, causes the clientcomputing device to send a second message to the computer system, wherethe second message includes the ontology-linked n-gram. The processincludes obtaining the second message from the client computing deviceafter an interaction with the UI to update the ontology. The processincludes updating the entry mapped to the ontology-linked n-gram andupdating a decision tree based on the update to the ontology.

Some aspects include a process that obtaining a set of natural-languagetext documents that discuss a topic, the set of documents containingdifferent states of knowledge about the topic at different times. Theprocess includes selecting an ontology from among a plurality ofontologies that correspond to different domains of knowledge, theselection being based on the ontology corresponding to a domain ofknowledge including the topic. The process includes identifying, usingthe ontology, concepts discussed in the documents. The process includesdetecting changes in at least some of the concepts over time based ondifferences between discussion of the concepts in documents authored atdifferent times. The process includes updating natural languageinstructions on the topic based on the detected changes in the conceptsand storing the updated natural language instructions in memory.

Some aspects include a tangible, non-transitory, machine-readable mediumstoring instructions that when executed by a data processing apparatuscause the data processing apparatus to perform operations including oneor more of the above-mentioned processes.

Some aspects include a system, including: one or more processors; andmemory storing instructions that when executed by the processors causethe processors to effectuate operations of one or more of theabove-mentioned processes.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned aspects and other aspects of the present techniqueswill be better understood when the present application is read in viewof the following figures in which like numbers indicate similar oridentical elements:

FIG. 1 is a schematic diagram of a first computing environment in whicha score stored in an account may be updated, in accordance with someembodiments of the present technique.

FIG. 2 depicts a logical and physical architecture of data stored in anontology model, in accordance with some embodiments of the presenttechniques.

FIG. 3 is a flowchart of an example of a process by which naturallanguage data may be converted in a set of ontology graphs, inaccordance with some embodiments of the present techniques.

FIG. 4 is a flowchart of an example of a process by which a query mayretrieve data based on a set of ontology graphs, in accordance with someembodiments of the present techniques.

FIG. 5 shows an example of a computer system by which the presenttechniques may be implemented in accordance with some embodiments.

FIG. 6 shows an example of different statement expansions based on aninitial statement, in accordance with some embodiments of the presenttechniques.

FIG. 7 shows an example of an initial query and an expanded query, inaccordance with some embodiments of the present techniques.

FIG. 8 shows the use of ontology graphs associated with differentclasses when determining an expanded query, in accordance with someembodiments of the present techniques.

FIG. 9 shows a representation of ontology graphs associated withdifferent classes of a hierarchical set of ontology graphs, inaccordance with some embodiments of the present techniques.

FIG. 10 is a flowchart of an example process by which a query may beexpanded based on a set of ontology graphs, in accordance with someembodiments of the present techniques.

FIG. 11 is a flowchart of an example process by which a hierarchical setof ontologies may be updated, in accordance with some embodiments of thepresent techniques.

FIG. 12 is a logical architecture indicating the integration of a datasystem with one or more learning systems, in accordance with someembodiments of the present techniques.

FIG. 13 is a flowchart of an example process by which a domain-specificsummarization may be provided based on a query, in accordance with someembodiments of the present techniques.

FIG. 14 is a flowchart of an example process by which a domain-basedsummarization model may be configured, in accordance with someembodiments of the present techniques.

FIG. 15 is an example user interface including an ontology-generatedsummary, in accordance with some embodiments of the present techniques.

FIG. 16 is a flowchart of an example process by which a query-augmentedindex is generated and used, in accordance with some embodiments of thepresent techniques.

FIG. 17 is a conceptual diagram of a workflow for generating orotherwise updating a query, in accordance with some embodiments of thepresent techniques.

FIG. 18 is a logical architecture indicating data flow through a dataingestion system, ontology-based language system, domain datasets, andinformation retrieval system, in accordance with some embodiments of thepresent techniques.

FIG. 19 is a flowchart of operations to for updating a user interfacefor displaying text of a document, in accordance with some embodimentsof the present techniques.

FIG. 20 is a flowchart of operations to for updating a user interfacefor updating a workflow, in accordance with some embodiments of thepresent techniques.

FIG. 21 is a diagram of an example set of user interface elementsindicating ontology-linked n-grams, in accordance with some embodimentsof the present techniques.

FIG. 22 is a diagram of an example set of user interface elementsindicating comparisons between different versions of a document, inaccordance with some embodiments of the present techniques.

FIG. 23 is a diagram of an example user interface displaying arepresentation of a decision tree, in accordance with some embodimentsof the present techniques.

FIG. 24 is a diagram of an example set of user interface elementspermitting the updating of a set of corpus and data processing elements,in accordance with some embodiments of the present techniques.

While the present techniques are susceptible to various modificationsand alternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Thedrawings may not be to scale. It should be understood, however, that thedrawings and detailed description thereto are not intended to limit thepresent techniques to the particular form disclosed, but to thecontrary, the intention is to cover all modifications, equivalents, andalternatives falling within the spirit and scope of the presenttechniques as defined by the appended claims.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

To mitigate the problems described herein, the inventors had to bothinvent solutions and, in some cases just as importantly, recognizeproblems overlooked (or not yet foreseen) by others in the field ofnatural language processing. Indeed, the inventors wish to emphasize thedifficulty of recognizing those problems that are nascent and willbecome much more apparent in the future should trends in industrycontinue as the inventors expect. Further, because multiple problems areaddressed, it should be understood that some embodiments areproblem-specific, and not all embodiments address every problem withtraditional systems described herein or provide every benefit describedherein. That said, improvements that solve various permutations of theseproblems are described below.

Various search systems are capable of retrieving information in responseto a query. However, while such systems may rank the relevance ofretrieved data based on a number of matches with exact terms or metadatatags, such operations may be less useful when a user is generatingqueries with incomplete information or limited expertise. Furthermore,the usefulness of any retrieved information may be limited if a user'schoice of words, choice of phrasing, or the context of the query itselfis not taken into consideration. Retrieving meaningful information for aquery under these conditions may require a different set of retrievaloperations, where such operations may fall under the field of naturallanguage understanding (NLU).

Some embodiments may address this issue by generating ontology graphsarranged in a hierarchical ontology data model based on ingesteddocuments. Ontology graphs may be associated with their own domaincategories or may be arranged into subgraphs having vertices associatedwith specific domain categories. Some embodiments may obtain a pluralityof documents and a corresponding set of domain vectors, where the set ofdomain vectors may be sent directly via an Application Program Interface(API), provided via a user interface (UI) element, determined from otherinformation, or the like. Such other information can include thedocument's origin, metadata associated with the document, a data format,or the like. A domain vector for a document can indicate various typesof information relevant to the usefulness of the document, such as anassociated expertise level for each of a plurality of domains, a countof words, a count of words having more than a specified number ofsyllables, or the like. Some embodiments may then use one or moremachine learning models to determine learned representations, such ascategories, scalar values, or embedding vectors, for the documents. Themachine learning models may include a transformer neural network modelsuch as Elmo, BERT, or the like. In some embodiments, the machinelearning model may improve data ingestion accuracy by generatingattention vectors for n-grams of an ingested document when performingdocument analysis or text summarization operations.

It should be understood that a set of ontology graphs may include asingle ontology graph, and that some embodiments may include multipledomains or classes within a single ontology graph. Various categoriesmay be used to categorize domain-related properties of an ontology graphand may be identified by a one or more domain category values. A domaincategory value may include a domain of knowledge (“domain”), asub-domain of a domain, a class within a domain, or the like, where anontology graph may be characterized by one or more domain categoryvalues. For example, an ontology graph may be characterized with adomain “cardiology” and a class value “3,” where the domain and theclass value identifying the domain of knowledge may be domain categoryvalues. Furthermore, while this disclosure may refer to a plurality ofontology graphs having different associated domains or domain classeswithin the domains, some embodiments may perform one or more operationsdescribed in this disclosure with one ontology graph. For example, someembodiments may use an ontology graph having different subgraphs, thedifferent subgraphs having an associated different set of domains orclasses within the domains. For example, some embodiments may store asingle ontology graph and its vertices, where different sets of ontologygraph vertices corresponding with different subgraphs of the ontologyAlternatively, or in addition, some embodiments may perform one or moreoperations of this disclosure using a single ontology graph that includemultiple vertices mapped to a same n-gram. For example, some embodimentsmay determine a first vertex and a second vertex of an ontology graphbased on a shared n-gram, where the shared n-gram may be mapped todifferent learned representations determined from the shared n-grambased on the surrounding n-grams of the shared n-gram. As used in thisdisclosure, an n-gram may map to a vertex of an ontology graph if anembedding vector of the n-gram or other learned representation of then-gram may be used to identify the vertex.

Some embodiments may identify a vertex of an ontology graph based on afirst embedding vector by matching the first embedding vector with a setof embedding vectors corresponding with a set of vertices of theontology graph. Alternatively, or in addition, some embodiments mayidentify a vertex of an ontology graph based on a first embedding vectorby determining a closest embedding vector in an embedding space withrespect to the first embedding vector and selecting the vertex mapped tothe closest embedding vector. Alternatively, or in addition someembodiments may identify a vertex of an ontology graph based on a firstembedding vector and a distance threshold. For example, some embodimentsmay determine a distance between a second embedding vector and the firstembedding vector in an embedding space and select the vertex mapped tothe second embedding vector based on the distance satisfying a distancethreshold. Furthermore, some embodiments may select a second vertexmapped to a second embedding vector based on the distance satisfying thedistance threshold and the second embedding vector being the closestembedding vector with respect to the first embedding vector.

Some embodiments may update an ontology graph based on the embeddingvectors or other learned representations representing words or othern-grams of the plurality of documents, where the ontology graph may beusable as an index for the plurality of documents. The ontology graphmay include vertices representing different n-grams, where the verticesmay be associated with each other via edges that indicate differentrelationships between the vertices or the documents with which they areassociated. A first word from one document may be associated with asecond word from a different document via an edge that categorizes therelationship between the first word and the second word. In someembodiments, the category may indicate a relationship based on the setof domain vectors associating a first word and a second word. Forexample, the category may reflect that a first word is a subcategory ofa second word, is associated with an expertise level greater than thesecond word, and is also part of a disambiguation group with the secondword. By updating an ontology graph with a set of domain vectors andusing the ontology graph as an index to retrieve documents, someembodiments may increase the speed of document retrieval and increasethe relevance of retrieved information by providing knowledge within thescope of a user's indicated expertise.

In some embodiments, an ontology data model or knowledge graph organizedby the ontology data model may improve the relevance of retrieveddocuments by accounting for a user's domain expertise or specificinterests. Such operations may be especially useful in specializedapplications where similar concepts may be disclosed in documents atdiffering levels of domain expertise, differing levels of securityclassification, or with differing amounts of relevance to subdomains.For example, if a user associated with a first hierarchical expertiselevel performs a search, some embodiments may obtain a first documentassociated with the first hierarchical expertise level and a seconddocument associated with a second hierarchical expertise level. Someembodiments may then provide a user with the document associated withthe first hierarchical expertise level. Additionally, by encodingrelative levels of domain expertise or other domain-specificrelationships in graph edges that indicate cross-domain relations, someembodiments may improve the speed and accuracy of responses to queriesfor information or provide other aspects of expert guidance. It shouldbe emphasized, though, that not all embodiments necessarily providethese advantages, as there are several independently useful ideasdescribed herein, and some implementations may only apply a subset ofthese techniques. As used in this disclosure, the term “ontology” may beused interchangeably with the term “ontology graph,” unless otherwiseindicated, where an entry of an ontology may include a vertex of theontology.

FIG. 1 is a schematic diagram of a first computing environment in whicha score stored in an account may be updated, in accordance with someembodiments of the present technique. In some embodiments, a computingenvironment 100 may be configured to mitigate some of theabove-described problems, such as challenges associated with retrievingdocuments based on queries. The computing environment 100 may include anetwork 150 in communication with a computer system 110 that receivemessages such as web requests or responses from a client computingdevice 104. As further discussed below, the client computing device 104may include kiosk terminals, virtual reality headsets, mobile computingdevices, laptops, desktop computers, tablet computers, or the like.

The client computing device 104 may be in a data session with thecomputer system 110 via the network 150, which allows the computersystem 110 to access data stored in a database(s) 132. The database(s)132 may include or otherwise be capable of accessing data stored in adocument database 134, an account database 136, or an ontology database138. As used in this disclosure, a database may refer to various typesof data structures, such as a relational database or a non-relationaldatabase. The computer system 110 may include servers stored in acentralized location, a cloud server system, a distributed computingplatform using different components or services, or the like. As furtherdescribed in this disclosure, records may include links to associatedrecords with respect to each other. In some embodiments, each of thedatabases may include data obtained from messages provided by externalcomputer systems, such data indicating a pre-generated ontology graph.In some embodiments, databases may persist program state to a media thatcan retain information even in the event that power is lost.Alternatively, or in addition, a database need not be persistent and caninclude in-memory databases, which can include non-persistent programstate.

In some embodiments, the computer system 110 may use ontology dataobtained from the ontology database 138 to retrieve a set of documentsfrom the document database 134 in response to a query provided by theclient computing device 104. Some embodiments may retrieve the set ofdocuments based on keywords, n-grams, word vectors, or the like. Inaddition to data encoded in the query, some embodiments may use datafrom the account database 136 to retrieve documents and sort the set ofretrieved documents. Furthermore, as described elsewhere in thisdisclosure, the ontology data stored in the ontology database 138 mayhave been obtained by the computer system 110 via the client computingdevice 104 or another data source, such as a centralized computingserver, a cloud server, or the like.

Some embodiments may store records of accounts, documents, ontologydata, or other data in a non-relational or distributed database such asApache Cassandra, MongoDB™, or the like. For example, some embodimentsmay store data in the document database 134 in the form of a Hadoopdatabase. Alternatively, or in addition, some embodiments may store datain a set of relational databases such as PostgreSQL™, Oracle my SQL™, orthe like.

Some embodiments may store an ontology graph or other graph data in adata structure exhibiting index-free adjacency, such as a labeledproperty graph or a resource description framework (RDF) model. Forexample, some embodiments may store an ontology graph in a graphdatabase model such as one used by Blazegraph, Janus graph, or Neo4j. Insome embodiments, using an implementation of an RDF graph model mayinclude adding a node to a graph portion template to include additionalinformation associated with the graph portion template. For example,some embodiments may update a first graph portion template to indicatethat a count of occurrences of the first graph portion template is nowequal to 193 by adding or otherwise updating a node of the first graphportion template in the Neo4j graph model to store the value “193” in anode titled “occurrenceCount.” In some embodiments, the data is storedin a graph database maintains index free adjacency to facilitaterelatively fast interrogation of the data structure, for instancewithout imposing a relatively large overhead from maintaining indexes,though embodiments are also consistent with use of other datarepositories, like relational databases, again which is not to suggestthat other descriptions are limiting.

As described above, various types of graph databases may be used, suchas Neo4j, DEX, Infinite Graph, or others described by Rawat et al.(Rawat, D. S. and Kashyap, N. K., 2017. Graph database: a complete GDBMSsurvey. Int. J, 3, pp. 217-226). In some embodiments, otherimplementations of a graph database may be used such as a Janus Graph™,Nebula Graph™, or the like. For example, some embodiments may build amodel of a graph portion template by applying a script to convert thegraph portion template into a Nebula Graph model, where the script mayprovide generate a query in the form of a graph-specific query languagesuch as nGQL. As discussed elsewhere in this disclosure, someembodiments may query the graph model using a graph-specific querylanguage such as nGQL or Cypher™.

For example, some embodiments may store ontology data in a set of SQLtables of the ontology database 138, where each record of the SQL tablemay represent a vertex record and include, as table fields, parentvertex identifiers, child vertex identifiers, categories indicatingrelationship types between vertices, scores associated with therelationship category, or the like. Some embodiments may store data in acombination of relational and non-relational databases. For example,some embodiments may store documents in a non-relational database andontology data in a relational database. In some embodiments, a record ofa relational or non-relational database may store a pointer, map, orother value usable for indicating relationships between a documentrecord, ontology data record, an account record, or other data.

As further discussed in this disclosure, some embodiments may performoperations to retrieve documents based on a query sent to the computersystem 110, where the documents may be selected based on one or moredomain indicators associated with the query. In some embodiments, thedomain indicator may be provided with a query or determined from thequery. Alternatively, or in addition, the domain indicator may beretrieved from a user account stored in an account database 136 orotherwise determined from context parameters associated with a datasession. Reference to “a record” followed by a reference to “the record”is consistent with scenarios where the record has changed in some regardbetween when the item is referenced, i.e., use of the indefinite articlefollowed by the definite article should not be read to suggest that thething referenced is immutable. Similar principles of construction shouldbe applied to other mutable entities, such as a user account, anontology or ontology data (e.g., a knowledge graph organized by theontology data model), or the like.

As discussed above, some embodiments may use inputs provided by a userto perform semantic searches. In some embodiments, using one or more ofthe operations described in this disclosure may provide search resultsthat match or exceeds other language models in general language tasks.For example, some embodiments may achieve 85-95% precision when testedusing the SQUAD 1.1 dataset or Quora duplicate questions dataset. Someembodiments may surpass other language models when used to performsearches in domain-specific tasks. For example, some embodiments mayachieve a 10 to 100% improvement in question-answer retrieval precisionin a role-specific domain based on the role's association with specificclasses of information associated with domain-specific knowledge. Someembodiments may include domain-specific operations or terminology,relationships, or contexts that may be relevant in only one domain or asmall number of domains. For example, some embodiments may relate theterms of a drug and an internal product code to each other andcategorize them as being associated with a shared concept (e.g., a sameanticoagulant name), even if other AI systems or NLP systems do notinclude these terms. As used in this disclosure, a concept may berepresented by a first vertex, a label or another type of categoryvalue, or the like. A concept may be or otherwise include a domaincategory value, where the subdomain represented by a concept may includethe vertices associated with the first vertex via a set of graph edges.

FIG. 2 depicts a logical and physical architecture of data stored in anontology model, in accordance with some embodiments of the presenttechniques. In some cases, some or all of the techniques described inthis disclosure may be implemented in the logical and physicalarchitecture 200. The client computing device 202 may send a query 204to a computer system 250. Data sent in the query 204 from the clientcomputing device 202 may include query text or terms used to retrievedocuments. In some embodiments, the query 204 may include or otherwisebe associated with session data, such as an account identifier, ausername, an indicated set of domain indicators, a feature associatedwith a user, or the like. In some embodiments, the session data may beprovided as a list of context parameters, a vector of values, or thelike. As further discussed below, some embodiments may expand a querybased on an ontology graph to increase the effectiveness of a semanticsearch.

In some embodiments, the query, or a response to the query, may be sentin form of a web message or a set of web messages. A “web message” is anapplication-layer communication over a network to or from a web browser(which may include a webview object in a native application, a headlessbrowser, or a browser extension). Web messages are not limited torendered content or user inputs, and web messages may be encoded inhypertext transport language protocol (HTTP, like HTTP2) or according toother application-layer protocols. A “web message” (expressed assingular) can include one or more transmissions, in some cases withintervening responsive messages, like acknowledgments or API responses.

Some embodiments may use various types of data to generate or otherwiseupdate the ontology data stored in the ontology data repository 230. Thedata may include a set of existing ontology data 211, a set ofnatural-language text documents 212, or a set of structured data 214.For example, the set of existing ontology data 210 may include anexisting knowledge graph structured in an existing ontology data model,such as the unified medical language system (UMLS) metathesaurus (MeSH).The existing knowledge graph may be stored in various ways, such as in arelational data structure, and may be imported into the ontology datarepository 230. As further discussed in this disclosure, different datatypes may be combined to update an ontology data model, such as onestored in an ontology data model record 231. The ontology data modelrecord may store values for record fields such as object categories,relationships between the categories, directional indicators of therelationships, or the like. Alternatively, or in addition, the ontologydata model may be stored in a knowledge graph such as a knowledge graph232, which may be formatted in a specified ontology data model. Forexample, the knowledge graph 232 may be stored as a set of recordsindicating that the knowledge graph is structured in a data modelspecified by an ontology data model record 231.

Some embodiments may store documents from the set of natural-languagetext documents 212 or set of structured data 214 into the documentsrepository 240. The set of natural-language text documents 212 may beobtained from various types of sources, such as an application programinterface (API) of a government server, an online textbook, a webpage,text stored in another database, or the like. For example, the documentsrepository 240 may include medical text from a textbook, legal text fromhealthcare law databases, internal medical record notes specific to apatient, or the like. As further discussed elsewhere in this disclosure,the documents of the documents repository 240 may be indexed by orotherwise accessed via data stored in the ontology data repository 230.For example, a pointer to a document stored in the documents repository240 may be stored in a vertex record of a knowledge graph stored in theontology data repository 230.

Some embodiments may store user account data in an account repository251, such as an account name, domain information, past activity, or thelike. For example, the account repository 251 may include a set of userrecords, each of which includes a username of an account and a set ofdomain indicators (e.g., categories, quantitative values, Booleanvalues, arrays, or the like) associated with the user. The set of domaincategories may indicate roles or knowledge domains of the user, such as“doctor,” “cardiologist,” “IT architecture,” or the like. In someembodiments, the knowledge domains may be associated with a category orother value indicating an expertise score. For example, a first useraccount record may include a knowledge category “cardiology” and anexpertise score “2” and a second user account record may include aknowledge category “cardiology” and an expertise score “5.” As discussedelsewhere in this disclosure, a categorical or quantitative scoreassociated with a domain, such as an expertise score, may change whichdocuments are presented to the client computing device 202 from thedocuments repository 240.

The processes presented in this disclosure are intended to beillustrative and non-limiting. In some embodiments, for example, themethods may be accomplished with one or more additional operations notdescribed or without one or more of the operations discussed.Additionally, the order in which the processing operations of themethods are illustrated (and described below) is not intended to belimiting. In some embodiments, the methods may be implemented in one ormore processing devices (e.g., a digital processor, an analog processor,a digital circuit designed to process information, an analog circuitdesigned to process information, a state machine, or other mechanismsfor electronically processing information). The processing devices mayinclude one or more devices executing some or all of the operations ofthe methods in response to instructions stored electronically on anelectronic storage medium. The processing devices may include one ormore devices configured through hardware, firmware, or software to bespecifically designed for execution of one or more of the operations ofthe methods.

FIG. 3 is a flowchart of an example of a process by which naturallanguage data may be converted in a set of ontology graphs, inaccordance with some embodiments of the present techniques. Operationsof the process 300 may begin at block 304. In some embodiments, theprocess 300 may include obtaining a corpus of text and an associated setof domain indicators, as indicated by block 304. The corpus of text mayinclude documents from various sources, where the text in the documentsmay be organized as a single text block or be separated into multiplesections of the document. Documents in the corpus may be separated inton-grams, where an n-gram may include a sequence of n items from text,where “n” represents an integer, and where the items may includephonemes, syllables, letters, words, symbols, base pairs, or the like.Different models may use different items as the base element for ann-gram. Additionally, a first n-gram does not need to include the samenumber of items as a second n-gram. For example, a first n-gram of afirst ontology graph vertex may be the word “verifying” and may bemapped to a second n-gram of a second ontology graph vertex, where thesecond n-gram may be the phrase “determining if a condition has beensatisfied.”

The corpus of text may be obtained from a variety of sources such assources available via the Internet, sources available via an internaldatabase or other data repository, information inputted into orotherwise provided via a UI element. For example, the corpus of text maybe obtained from a medical textbook, a financial statement, an onlinedatabase of medical information, a contract, a set of governmentregulations, or the like. In many cases, the corpus of text may includeunstructured natural-language text documents such as a textbook chapteror science paper, which may be contrasted with structured language textdocuments such as a table of values, an enumerated list of values, orthe like. Some embodiments may use data source profiles to determine orotherwise update a set of domain indicators associated with a documentobtained from the corresponding data source. For example, if a subset ofdocuments of a corpus is obtained from a company's data repository, someembodiments may retrieve a data source profile of the company andassociate previously-entered metadata stored in the data source profilewith the subset of documents. In some embodiments, the corpus mayinclude unstructured natural-language text, such as passages of text,video transcripts, or the like. Some embodiments may pre-process videoor audio to transform the same into text, for instance withspeech-to-text algorithms or scene description algorithms.

Various operations may be performed when obtaining a corpus for use aspart of a structured knowledge base that is usable as part of aknowledge fabric, which is further described below. In some embodiments,the corpus may include data generated from media files, such as metadataassociated with images, object recognition of videos, or the like. Thecorpus may include data obtained from an API of a database service(e.g., DropBox, Google Drive, Veeva Vault, DXC, Microsoft Dynamics,etc.). Some embodiments may inspect data being provided to the corpus toverify that the information encoded in the data is secure or accurate.For example, some embodiments may determine that data being added to acorpus includes a set of expected values or terms or determine thatmessages received from an API satisfies one or more web filters. Someembodiments may further ingest and distinguish between publiclyavailable data (e.g., data obtained by an organization from a datasource that is not controlled by the organization) and private data. Forexample, some embodiments may ingest publicly available data from agovernment entity and ingest private data stored as free text customerfeedback.

Some embodiments may obtain a set of domain indicators associated withthe corpus of text. Each document in a corpus of text or a subset ofdocuments in the corpus of text may store or otherwise be associatedwith a set of metadata tags indicating a domain of the document. Forexample, each respective document in a corpus of text may be store afirst respective domain indicator representing a domain categoryselected from a set of categories (e.g., “[cardiology, gastroenterology,gastronomy]”) and a second respective domain indicator indicating anexpertise score for the corresponding domain category (e.g., a numericvalue ranging between zero and ten). Various types of domains may beindicated, such as a specific document topic, a field of study discussedin the document, a target audience for the document, a user role havingpermission to read the document, or the like. In some embodiments,documents may be associated with categorical values or numerical valuesindicating a complexity or target expertise of a document. For example,some embodiments may obtain a document that is associated with a vector“[1, 5],” where the first element “1” of the vector may indicate aspecific domain, and the second element “5” of the vector may indicatean expertise score (e.g., “class”). Alternatively, or in addition, someembodiments may generate the vector or other list of values may be usedto indicate expertise for a variety of domains or derived domaincategories. For example, a list of values may include “[0, 5, 0, 0,20],” where each number of the list may represent a class for one of thefive different fields of domain knowledge. As further discussed below,some embodiments may use an indicated expertise score or other scoreassociated with a domain to determine a hierarchy or other order betweendifferent ontologies or knowledge graphs. A set of ontology graphsorganized in a hierarchy may be used as part of an index for a knowledgefabric, which may include a set of documents or other data, the ontologysystem(s) and indices used to organize the set of documents or otherdata, or the functions used use the ontology system(s) or indices usedto retrieve information from the set of documents or other data. Byusing a knowledge fabric that is organized by a set of ontology graphs,some embodiments may quickly navigate through different knowledgedomains or different classes within those domains to retrieve relevantqueries for a specific user.

The process 300 may include determining a learned representation ofn-grams based on the obtained corpus, as indicated by block 308. Alearned representation may include various value types, such ascategories, Boolean values, quantitative values, or the like. In someembodiments, a learned representation may include a set of embeddingvectors in a multi-sense embedding space. Some embodiments may determinea learned representation for each n-gram in a document, where thelearned representation may include an embedding vector, where theembedding vector may include a set of values in an embedding space thatindicate a position in the embedding space. Some embodiments maydetermine the embedding space by using a statistical method ormachine-learning method. Some embodiments may determine an embeddingvector in a multi-sense embedding space for an n-gram, where amulti-sense embedding space may allow the same n-gram to correspond withdifferent factors in an embedding space. For example, the n-gram may bea first word “run” in a document and may correspond with two differentembedding vectors in a multi-sense embedding space based on words aroundthe first word in the document.

As described elsewhere in this disclosure, some embodiments may performone or more machine-learning operations to determine a set of embeddingvectors in an embedding space. The embedding space of a word vector mayinclude a large number of vector dimensions, such as more than 10dimensions, more than 100 dimensions, more than 1000 dimensions, or thelike. In some embodiments, the embedding space used to represent ann-gram may have fewer dimensions than a cardinality of the n-grams. Forexample, a corpus may include over one million n-grams, over ten millionn-grams, over one hundred million n-grams, or the like. Some embodimentsmay represent n-grams of such a corpus with less than one hundredthousand dimensions, less than twenty thousand dimensions, less than tenthousand dimensions, or less than one thousand dimensions, or the like.

As described elsewhere in this disclosure, some embodiments may alsodetermine relationships between learned representations using a machinelearning operation. For example, some embodiments may use a trainedneural network to determine relationships between different n-grams orother values represented by ontology vertices based on first set ofn-grams represented by the ontology vertices and another set of n-gramssurrounding the first set of n-grams. In some embodiments, therelationships between different concepts, ontology vertices, or otherelements of an ontology may be encoded as ontological triple. Anontological triple may include a first value identifying a first vertex,a second value identifying a second vertex, and a third value thatcategorizes or quantifies a relationship between the first vertex andthe second vertex. For example, some embodiments may determine that afirst vertex representing the document-obtained n-gram “smartphone” hasa categorical relationship of “subset” with respect to a second vertexrepresenting the document-obtained n-gram “computing device.” In someembodiments, this relationship may be determined based on a sequence ofdocument-obtained n-grams, such as the phrase “is a type of.”

Some embodiments may generate a set of embedding vectors using a neuralnetwork model that determines an embedding vector for a first n-gramwithout using data based on the n-grams around the first n-gram, such asa continuous-bag-of-words (CBOW) model, Skip-gram model, or other modeldescribed in Bhoir et al. (Bhoir, S., Ghorpade, T. and Mane, V., 2017,December. Comparative analysis of different word embedding models. In2017 International Conference on Advances in Computing, Communicationand Control (ICAC3) (pp. 1-4). IEEE), which is hereby incorporated byreference. For example, some embodiments may perform a shallow neuralnetwork model, such as a Word2Vec model (which may use either of or boththe CBOW model and the Skip-gram model) to determine embedding vectorsfor words or other n-grams of a document.

Alternatively, or in addition, context-independent embedding operationsother than neural-network-based operations may be used, such as a matrixfactorization method. For example, some embodiments may use a GlobalVector (“GloVe”) model to determine an embedding vector for an n-gram,where using a GloVe model may include using a matrix model trained aglobal word to word co-occurrence count, and where the GloVe model maybe further described by Pennington et al. (Pennington, J., Socher, R.and Manning, C. D., 2014, October. Glove: Global vectors for wordrepresentation. In Proceedings of the 2014 conference on empiricalmethods in natural language processing (EMNLP) (pp. 1532-1543)), whichis incorporated herein by reference.

Some embodiments may use objects other than words as n-grams, such assub-words or groups of words. For example, some embodiments may use amodel that splits a word such as “apple” into the tri-gram “app,” “ppl,”and “ple,” where the word embedding vector for apple will be the sum ofthe n-grams. Various neural network models may be used to determineembedding vectors for multiple n-grams generated from one word, such asa FastText model or another shallow neural network model (i.e., a neuralnetwork having fewer than four hidden neural network layers). Forexample, some embodiments may use a shallow neural network model todetermine embedding vectors for the word “cardiomyopathy” by splittingthe word into the n-grams “cardio,” “myo,” and “opathy,” determining anintermediate vector for each individual n-gram, and determining anembedding vector based on the three intermediate vectors.

Some embodiments may determine an embedding vector associated with ann-gram using a model based on both the n-gram itself and the contextsurrounding the n-gram (e.g., other n-grams, syntax, semantics). Forexample, some embodiments may use neural networks models trained on aset of text of a corpus or other training data to predict n-grams basedon other n-grams in a system via a set of attention values for then-grams, where the attention values may be used to weigh or otherwisemodify an output of a neural network. Various models may use differenttypes of neural network models, perform different pre-processingoperations, use different operations to determine attention values, orthe like. For example, some embodiments may use bidirectional long shortterm memory (LS™) neural networks or another recurrent neural network togenerate encoding vectors, such neural networks described in Embeddingsfrom Language Models (ELMo), as described by Peters et al. (Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K. andZettlemoyer, L., 1802. Deep contextualized word representations. arXiv2018. arXiv preprint arXiv:1802.05365), which is hereby incorporated byreference. By determining embedding vectors or other learnedrepresentations of words or other n-grams based on their surroundingwords or n-grams, some embodiments may account for word or phrasedisambiguations.

Various methods may be used to determine attention values and use theattention values. For example, some embodiments may use a multi-headedattention-based autoencoder trained to use attention values mapped ton-grams determined with attention heads, such as autoencoders using amodel similar to those described by Vaswani et al. (Vaswani, Ashish,Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,Lukasz Kaiser, and Illia Polosukhin. “Attention is all you need.” InAdvances in neural information processing systems, pp. 5998-6008. 2017,arXiv:1706.03762) or Devlin et al. (Devlin, J., Chang, M. W., Lee, K.and Toutanova, K., 2018. Bert: Pre-training of deep bidirectionaltransformers for language understanding. arXiv preprintarXiv:1810.04805), which are incorporated herein by reference. Suchembodiments may use a multi-headed attention model, where an attentionhead of the multi-headed attention model may determine a set ofquery/key/value weight matrices for each attention head during training.In some embodiments, Q, K, and V may be determined as projectionvectors. For example, Q may represent a query vector indicating ann-gram position(s) in a sequence of n-grams and K or V may represent akey vector indicating all the n-gram positions in a sequence of n-grams.This plurality of weight matrices may then be used to determine aplurality of attention matrices, where each element of the attentionmatrices may represent a respective attention value of a respectiven-gram of a set of n-grams. The attention matrices may then beconcatenated and multiplied by a weights matrix based to determine anoutput set of attention values. Some embodiments may determine an outputset of attention values for each set of n-grams of a document todetermine attention values for the document. Additionally, someembodiments may include a position vector or other set of positionvalues to indicate the positions of n-grams in a sequence of n-gramsrelative to other n-grams of the sequence of n-grams. In someembodiments, a position vector may follow a specified pattern based onthe respective position of the respective n-gram relative to othern-grams from the same document, such as n-grams that are in the samesentence as the respective n-gram or the same paragraph as therespective n-gram. For example, each respective position value of aposition vector for a respective n-gram of a sequence of n-grams may bemonotonically increased with respect to its position amongst the othern-grams of the sequence of n-grams.

Some embodiments may compute positive features to determine attentionvalues used to determine an embedding vector. For example, someembodiments may use a model described by Choromanski et al.(Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A.,Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L. andBelanger, D., 2020. Rethinking Attention with Performers. arXiv preprintarXiv:2009.14794), which is herein incorporated by reference. Someembodiments may generate a pair of random feature maps using a featuremap function ϕ. Various types of the feature map functions may be used.For example, some embodiments may use the form shown below in Equation1, where h(x) may be a function of an input x, such as the input valueitself, where m and l represent dimension size, the functions f₁ tof_(l) or f_(m) may include one or more real functions (e.g., a sinfunction, cosine function, or the like), and ω₁ and ω₁ to ω_(m) mayrepresent a set of deterministic vectors obtained from a probabilitydistribution, and where the superscript T may indicate a transposeoperation:

$\begin{matrix}{\phi = {\frac{h(x)}{\sqrt{m}}\left( {{f_{1}\left( {\omega_{1}^{T}x} \right)},\ldots\mspace{14mu},{f_{1}\left( {\omega_{m}^{T}x} \right)},{\ldots\mspace{14mu}{f_{l}\left( {\omega_{1}^{T}x} \right)}},\ldots\mspace{14mu},{f_{l}\left( {\omega_{m}^{T}x} \right)}} \right)}} & (1)\end{matrix}$

Some embodiments may reduce the variance of an estimated value, such asan approximated key vector or an approximated query vector, byentangling random samples that are orthogonal by using a Gram-Schmidtrenormalization process. Some embodiments may generate a random featuremap ϕ(x) based on an input vector x, a second random feature map ϕ(y)based on the input vector y, and determine an approximated key matrixbased on the inner product of first and second random feature maps (i.e.“ϕ(x)^(T) ϕ(y)”), such as an approximated key matrix equal to theexponential of the inner product (i.e. “E[(x)^(T) ϕ(y)]”). Someembodiments may determine a set of attention values by determining anapproximated query vector Q′ equal to ϕ(Q), an approximated value vectorK′ equal to ϕ(K), and determine a set of attention values based on theapproximated key matrix, and where Q and K may be original query and keyvectors determined as projection vectors based on the input sequence ofn-grams. For example, some embodiments may determine the set ofattention values for a sequence of n-grams of a document described inthis disclosure using implementations of Equation 2 and 3, where

may represent the set of approximated attention values, V represents avalue vector, {circumflex over (D)} may represent an approximateddiagonal function, and 1_(L), may represent an identity matrix of sizeL, where L may indicate a size of the sequence of n-grams:

={circumflex over (D)} ⁻¹(Q′(K′)^(T) V)  (2)

{circumflex over (D)}=diag(Q′((K′)^(T)1_(L))  (3)

The process 300 may include updating vertices of a set of ontologygraphs based on the set of embedding vectors, as indicated by block 312.As described elsewhere in this disclosure, some embodiments may store orotherwise access ontology data such as knowledge graphs written in anontology data model or the structure of the ontology data model itself.As used in this disclosure, the term ontology graph may refer to eitheror both an ontology data model and a knowledge graph stored in theformat of the ontology data model. As used in this disclosure, a vertexof an ontology graph may be associated with a learned representation byhaving the learned representation as an identifier, a hash value of thelearned representation as an identifier, storing the learnedrepresentation or hash value of the learned representation in a relatedrecord, or the like. A vertex of a graph may be stored as a singlerecord of a database, a set of different records or fields of therecord, or the like. For example, a vertex associated with a learnedrepresentation of an n-gram may include a first record stored in a firstdatabase and a second record stored in a second database. The firstrecord may include a graph identifier of the first vertex and anidentifier to the second record, and the second record may include thelearned representation of the n-gram.

As discussed elsewhere in this disclosure, some embodiments may obtain aset of ontology graphs from an initial source or generate the ontologygraph from a set of initial data. For example, some embodiments mayobtain a set of ontology graphs, including a graph representing anontology data model and a knowledge graph formatted in the ontology datamodel from a medical repository, government repository, commercialrepository, or the like. Some embodiments may update a knowledge graphformatted in the form of an ontology data model or another ontologygraph based on the set of embedding vectors determined using one or moreoperations described in this disclosure. For example, a knowledge graphmay include a vertex identified or otherwise associated with a firstembedding vector, where a vertex may correspond with an embedding vectoror other learned representation if the learned representation is anidentifier of the vertex or is stored in a record of the vertex.

Some embodiments may update a vertex or data associated with the vertexbased on the set of embedding vectors by updating a set of storedpointers to documents to indicate a document storing the n-gram,updating a count of the number of n-grams being used, or the like. Forexample, an embodiment may determine that the n-gram “blue” is used in afirst document based on the set of n-grams determined using one or moreoperations described in this disclosure. In response, the embodiment maystore a pointer to the first document and a count of the times then-gram is used in the first document. Some embodiments may use thisinformation to rank or otherwise select one or more documents from acorpus based on a query.

Some embodiments may generate or update a plurality of ontology graphsbased on the set of learned representations. Additionally, someembodiments may obtain a plurality of initial ontology graphs and updateeach of the initial plurality graphs as new data is received oranalyzed. For example, some embodiments may obtain a first ontologygraph storing embedding vectors representing aeronautical engineeringconcepts and a second ontology graph storing embedding vectorsrepresenting airplane pilot concepts. Some embodiments may then obtain acorpus of text and independently update each of the ontology graphs withadditional vertices based on the embedding vectors of n-grams of thedocuments of the corpus using one or more operations described above. Insome embodiments, relationship types between already-encountered learnedrepresentations and newly learned representations may be also be learnedor interpreted using a machine learning method or statistical method.For example, a neural network may be trained to determine that a firstlearned representation is a conceptual subset of a second learnedrepresentation based on the detection of a phrase “is a type of” or “isone of” Alternatively, or in addition, some embodiments may includeoperations to use structured data, such as tabular data, to determineassociations between vertices. For example, a row of an imported datatable may indicate that a first n-gram may be equivalent to a secondn-gram, and this indication may be used to generate an associationbetween a first vertex corresponding to the first n-gram and a secondvertex corresponding to the second n-gram.

The process 300 may include determining a set of vertex groups based onthe set of ontology graphs, as indicated by block 320. A vertex groupmay include a set of multiple vertices or an aggregation of the multiplevertices of an ontology graph and may be categorized or otherwiseclassified based on the types and methods used to group its vertices. Avertex group may include vertices representing derived values computedfrom learned representations. For example, a vertex group may include avertex representing a centroid of the vectors. In addition, someembodiments may determine other values derived from a group of vertices,such as a set of values or functions representing a boundary of thevectors surrounding vertices of the vertex group.

Some embodiments may use an unsupervised learning operation to map oneor more concepts represented by vertex groups to n-grams. For example,some embodiments may determine a vertex group using a clustering method,such as a K-means clustering method or a hierarchical clustering method,to determine a vector cluster. Each respective vector of the vectorcluster corresponds with a respective vertex of the vertex group.Vertices of a vertex group may be described as being of the same clusterif their corresponding vectors are assigned to the same cluster during aclustering operation. For example, some embodiments may use a K-meansclustering method after determining an initial set of centroids ofvectors in a multi-sense embedding space. Some embodiments may determinethe initial set of centroids based on an initial knowledge graph,determine a set of neighboring vertices of the centroid based on a setof pairwise distances between the set of neighboring vertices and thecentroid in the embedding space, and re-compute each of the respectivecentroids based on the set of neighboring vertices. The use of theK-means clustering method may provide a fast way of determining groupsof vertices and their associated n-grams.

Some embodiments may determine a vertex group using a density-basedclustering method, such as using an implementation of a density-basedspatial clustering of applications with noise (DBSCAN) algorithm. Forexample, some embodiments may use DBSCAN algorithm implementation thatfinds the neighboring vectors of a first embedding vector assigned to afirst cluster. Using the DBSCAN algorithm implementation may alsoinclude finding the vectors within a threshold distance of the corevector and assign the vectors to the first cluster or otherwiseassociate the vectors with the first cluster. For example, someembodiments may determine that a plurality of pairwise distances betweena first vector and a plurality of other vectors is less than a distancethreshold, where the vectors may be in a multi-sense embedding spacedescribed in this disclosure, and where the plurality of pairwisedistances may be in the same multi-sense embedding space. Someembodiments may then check that a count of the plurality of the othervectors satisfies a minimum vectors threshold, such as at least twoother vectors, at least five other vectors, or the like. Someembodiments may then associate each respective vector of the pluralityof other vectors with the vector cluster. Some embodiments may theniteratively perform these steps until no further assignments orre-assignments to a cluster occurs. In some embodiments, a determinationthat a pair of vector are part of a same cluster may indicate a degreeof semantic similarity between the n-grams represented by the pair ofvectors, where a lesser distance may be correlated with an increaseddegree of semantic similarity.

Some embodiments may categorize a vertex group determined from aclustering method as a first type of vertex group, where vertices of avertex group of the first type of vertex group may be associated withvectors categorized as being part of a same cluster. In someembodiments, the vertex group may represent a ‘concept’ in a domain,where the concept may be shared amongst multiple classes of the domain.Alternatively, or in addition, the vertex group may represent a‘concept’ for a specific class of the domain. As described elsewhere inthis disclosure, one or more n-grams may be mapped to a plurality ofdifferent vectors, two or more of which may be mapped to differentclusters. For example, a first n-gram may be mapped to a first vectorand a second vector, where the first vector may be part of a firstcluster representing a first concept and where the second vector may bepart of a second cluster representing a second concept.

Some embodiments may update a knowledge graph based on a vertex groupdetermined using a clustering operation. For example, some embodimentsmay determine that an embedding vector is closest to a centroid of acluster of vectors in a multi-sense embedding space and, in response,update the vertex of an ontology graph corresponding with the embeddingvector to indicate that the embedding vector is near the centroid.Alternatively, or in addition, some embodiments may generate or update avertex of an ontology graph based on the centroid of the cluster, wheresome embodiments may indicate that the vertex represents a derived setof values.

The process 300 may include determining a set of hierarchicalrelationships for the set of ontology graphs, as indicated by block 324.As described elsewhere in this disclosure, knowledge graphs or otherontology graphs may be organized in different domains or sub-domains. Insome embodiments, a set of ontology graphs may be sorted into ahierarchy of ontology graphs based on domains associated with thevertices of the ontology graph and categories or quantitative valuesassociated with the domains.

Some embodiments may determine a hierarchy of ontologies based on one ormore vertex groups associated with a vector cluster via an edgeconnection between the vertex group and one or more shared connections.Some embodiments may determine that a first vertex group of a firstontology graph may be associated with a vertex of a second ontologygraph via one or more shared vertices. For example, a first vertex groupmay include a set of vertices corresponding with a first set of learnedrepresentations that includes the embedding vector [x₁, x₂, x₃]. Someembodiments may determine that the embedding vector corresponds with avertex in a second ontology graph, and, in response, determine that thefirst vertex group is associated with the vertex in the second ontologygraph. Additionally, if the vertex in the second ontology graph is partof a second vertex group, some embodiments may determine that the firstvertex group is associated with the second vertex group of the ontologygraph.

Some embodiments may then determine a hierarchy between the firstontology graph and the second ontology graph based on a relationshipbetween the first vertex group and the second vertex or second vertexgroup. For example, some embodiments may determine that a first vertexof a first ontology graph is associated with a first domain indicatorrepresenting a first expertise score, and that the first vertex is partof a first vertex group. The first ontology graph may be part of a setof ontology graphs that includes a second ontology graph, where thesecond ontology graph may include a second vertex. The second vertex maybe associated with a second domain indicator representing a secondexpertise score. In response to a determination that the first vertexgroup is associated with the second vertex, some embodiments may thendetermine a hierarchy order between the first ontology graph and thesecond ontology graph based on the first domain indicator and the seconddomain indicator. For example, if the first vertex is associated with adomain expertise score of “1,” indicating basic expertise, and if thesecond vertex is associated with a domain expertise score of “10,”indicating extreme expertise, some embodiments may determine that thefirst ontology graph is lower on a domain hierarchy than the secondontology graph. Alternatively, or in addition, some embodiments maydetermine a hierarchy of concepts based on a shared n-gram between twocluster of vectors corresponding to two different concepts. For example,a first n-gram may be mapped to a first vector and a second vector,where the first vector may be part of a first cluster representing afirst concept and where the second vector may be part of a secondcluster representing a second concept.

Some embodiments may determine that the first and second cluster sharean n-gram and, in response, determine a hierarchy between the twoconcepts based on the documents associated with the concepts. Forexample, a first and second ontology graph may be associated with thedomain “medical billing,” where documents associated with vertices ofthe first ontology graph may be associated with the class category“expert,” and where documents associated with vertices of the secondontology graph may be associated with the class category “layman.” Someembodiments may then associate the first ontology graph with thecategory “expert” and the second ontology graph with the category“layman” based on their associated class categories and update ahierarchy of the two ontology graphs such that the first ontology graphhas a higher hierarchy value than the second ontology graph. In someembodiments, the classes or other categories used may be mutuallyexclusive with respect to each other. For example, an ontology graphlabeled as “expert” may be forbidden from being labeled as “layman,”where the ontology graph may be labeled with categories from a set ofmutually exclusive categories such as, “layman,” “beginner,” “advanced,”and “expert.” Some embodiments may account for the user's goals, forinstance, determining a hierarchy based on whether the user intends toengage in a higher-risk activity in which more certainty is warrantedthan in a lower-risk activity, or based on whether the user intends toexplore new relationships between concepts or determine which are wellestablished relationships.

In some embodiments, a first and second ontology graph may be associatedwith computed domain indicators, such as a computed expertise score. Forexample, vertices or other subgraph components of a first ontology graphmay be associated with a first set of documents, and vertices or othersubgraph components of a second ontology graph may be associated with asecond set of documents. Some embodiments may determine a measure ofcentral tendency of the first set of expertise scores of the first setof documents, such as a mean average of the first set of expertisescores and a measure of central tendency of the second set of expertisescores of the second set of documents, such as a mean average of thesecond set of expertise scores. Some embodiments may then determine ahierarchy between the first ontology graph and the second ontology graphbased on the measures of central tendency.

The process 300 may include alerting data providers or data monitorsbased on a set of criteria associated with a set of documents, a set ofvertices, a set of vertex groups, or the hierarchy of the set ofontology graph relationships, as indicated by block 330. Various alertsmay be generated based on possible discrepancies or new domainsdetermined from the set of ontology graphs or their associateddocuments. The alerts may indicate that a document should be associatedwith a different domain indicator, that a new domain may exist based ondifferent users viewing a same set of documents, that a new ontologygraph should be generated, or the like.

Some embodiments may determine a predicted domain indicator for adocument using a prediction model, such as a neural network or astatistical method. Some embodiments may determine a vector representingexpertise scores for one or more documents using a neural networktrained to predict a document's complexity based on the n-grams of thedocument. For example, an encoder neural network may be used to predicta domain indicator such as a single domain category, an expertise scoreassociated with the single domain category, a plurality of domaincategories, a plurality of expertise scores corresponding with aplurality of domain categories, or the like. After obtaining a trainedencoder neural network, some embodiments may then determine a predictedset of domain indicators for an obtained document and then determinewhether the predicted set of domain indicators is different from theobtained set of domain indicators associated with the obtained document.In response to a determination that the predicted set of domainindicators differs from the obtained set of domain indicators, someembodiments send an alert to a client computing device or anothercomputing device of a data-providing entity or a data-monitoring entity.Alternatively, or in addition, some embodiments may use a trained neuralnetwork to determine a predicted domain indicator in the form of aquantitative value and then determine whether the predicted domainindicator is within a threshold range of the corresponding obtaineddomain indicator associated with the obtained document. In response to adetermination that the predicted domain indicator exceeds the thresholdrange of the obtained domain indicator, some embodiments may send analert to a client computing device or another computing device of adata-providing entity or a data-monitoring entity. For example, someembodiments may sent an alert indicating a mismatch between a predictedset of domain indicators for a document and an obtained set of domainindicators for the document.

Some embodiments may determine that an n-gram is mapped to a pluralityof concepts and, in response, send an alert to a data provider or datamonitor requesting clarification or additional information. For example,some embodiments may determine that the n-gram “hemorrhage” isassociated with both a first embedding vector and a second embeddingvector. The first embedding vector may be a vector of a first clusterassociated with a first concept in a medical domain, and the secondembedding vector may be a vector of a second cluster associated with asecond concept in a financial domain. Some embodiments may then select adata provider listed as an expert in at least one of the first medicaldomain or the financial domain based on an expert score associated withthe data provider, where expertise may be associated with a specific setof categories or specific set of values. Alternatively, or in addition,expertise may be associated with having an expertise score thatsatisfies an expertise threshold. Some embodiments may then send analert to the data provider categorized as an expert to request an inputto characterize at least one of the first concept or the second concept.Some embodiments may characterize a concept by providing a textdefinition of the concept, determining the boundaries in an embeddingspace associated with the concept, determining the embedding vectors inembedding space associated with the concept, confirming that anassociation between the n-gram and the concept is valid, providing analternative cluster or additional cluster for a concept, or the like.

Some embodiments may determine a first set of accounts associated with afirst set of domain indicators and a second set of accounts associatedwith a second set of domain indicators, where users corresponding toboth the first and second sets of accounts frequently access the sameset of documents. In response, some embodiments may determine whetherthe first set of accounts and a second set of accounts share a subset ofdomain indicators between the first set of domain indicators and asecond set of domain indicators. For example, a first set of users maybe labeled or otherwise associated with a first vector in an accountparameter space, and a second set of users may be labeled or otherwiseassociated with a second vector in the account parameter space. Anaccount parameter space may include dimensions representing variousvalues such as a domain parameter space, other account parameters, orother parameter values. For example, an account parameter space mayinclude a set of domain indicators such as domain categories,quantitative values representing expertise scores in their respectivedomain categories, demographic information such as education statuses,system use information such as a history of previously-accessedarticles, or the like. Some embodiments may generate an indicator of theproposed set of accounts, generate an alert that a possible new domainvector or other proposed new domain indicator has been detected, ordirectly generate the new domain vector as a proposed new domainindicator to be associated with one or more accounts.

The process 300 may include updating an index based on the set ofontology graphs, as indicated by block 340. As described elsewhere inthis disclosure, a set of documents stored in a repository may beaccessed via one or more pointers stored as a part of or otherwiseassociated with a vertex of an ontology graph. For example, a knowledgegraph may include a set of vertices corresponding to embedding vectors.Each vertex may store or be associated with one or more documents orpositions within the document using the n-gram associated with thevertex.

As described elsewhere in this disclosure, the knowledge graph or otherontology graph data determined using one or more of the operations ofthis disclosure may be used as an index, where updating the index may beperformed by updating the ontology graph. Alternatively, or in addition,some embodiments may include an index that is independent of a knowledgegraph and may cause the update of the index by updating thecorresponding knowledge graph. For example, a first knowledge graph mayinclude pointers to a first set of records of an index. In response to adetected update to the first knowledge graph causing the association ofan additional document with a vertex of the knowledge graph, someembodiments may update the corresponding index to include an additionalrecord pointing to the additional document.

Various types of indices may be constructed or updated based on to anontology graph, such as an index having a self-balancing tree datastructure (“B-tree index”), where a b-tree index may include a set ofindex nodes starting at root index node. A B-tree index may have anorder value m, where each index node has at most m child index nodes,each non-leaf index node has at least m/2 index nodes, and a non-leafindex node having k child index nodes will contain a proportional numberof keys for their child index nodes. An index node of a B-tree mayinclude a key value and a pointer to another index node. In someembodiments, the key value of the index node may correspond to one of apair of n-grams, where a child index node of the index node acting as aleaf index node may include a pointer to or other identifier of theother n-gram of the pair of n-grams. Alternatively, or in addition, someembodiments may store data other than a pointer or identifier in a leafindex node, such as a text summary or an entire document.

Some embodiments may store an association between a pair of concepts,pair of vertices, pair of n-grams, or pair of embedding vectors in anindex. Some embodiments may also store a related categorization orquantification of the association, such as a difference between classvalues, in the index. For example, some embodiments may include orassociate a difference associated with a graph edge between a firstn-gram and a second n-gram that indicate class value difference of “1”between the first n-gram and the second n-gram. In some embodiments, theclass value difference “1” may indicate that the first n-gram isassociated with a class value that is greater than the class value ofthe second n-gram by “1.” The value may be stored in various ways, suchas directly in a leaf node of an index stored in a B-tree structure, ina record identified by an index node, or the like.

Some embodiments may increase the utility of the index by updating theindex to include references between documents based on a hierarchy ofontology graphs. For example, some embodiments may index a first set ofdocuments using a first set of index records based on a first ontologygraph. The first set of graph-indexed documents may then be updatedduring or after a determination that the first ontology graph is greateron a hierarchy with respect to a second ontology graph, where the secondontology graph comprises a vertex associated with a second document thatis not in the first set of graph-indexed documents. Some embodiments maythen determine that the first set of documents has a greater domainindicator value than the second document based on the hierarchy of theontology graph. Alternatively, the first set of graph-indexed documentsmay then be updated during or after a determination that the firstontology graph is lesser on a hierarchy with respect to the secondontology graph, where the second ontology graph comprises the vertexassociated with the second document not in the first set ofgraph-indexed documents. Some embodiments may then determine that thefirst set of documents has a lesser domain indicator value than thesecond document based on the hierarchy of the ontology graph.

Additionally, the index may further be updated to indicate documentsrelated to each other via vertex adjacency. For example, someembodiments may determine that a first vertex corresponding to a firstlearned representation of a first n-gram is associated with a secondvertex corresponding to a second learned representation of a secondn-gram, where the first vertex and second vertex vertices of differentontology graphs, and where the first vertex is associated with a firstdocument, and where the second vertex is associated with a seconddocument. Some embodiments may then determine that the second vertex isassociated with a third vertex adjacent to the second vertex. Variousoperations may be performed to determine that two vertices of anontology graph are adjacent. For example, some embodiments may determinethat the adjacent vertex is associated with the second vertex based on apre-existing edge associating the second vertex with the third vertex.Alternatively, or in addition, some embodiments may associate the secondvertex with the third vertex by generating an edge between the twovertices in response to a determination that n-grams of a corpusassociate the two corresponding n-grams of the two vertices.Alternatively, or in addition, the edge may be generated in response toa clustering operation such as one described elsewhere in thisdisclosure.

Based on the association between the first vertex with the second vertexand the second vertex with the third vertex as described above, someembodiments may generate an edge or other encoded association betweenthe first vertex and the third vertex. In some embodiments, this thirdvertex may be associated with a third document not associated with thefirst vertex or the second vertex. In response to an association betweenthe first vertex and the third vertex, some embodiments maycorrespondingly update a record of the first document in an index toinclude a pointer or other reference to the third document. In addition,some embodiments may update a hierarchical relationship between thefirst document and the third document, which may increase the speed ofdocument retrieval. Various other associations in the index may be made,such as associating the first vertex with the second document or thethird document. By associating documents of a corpus in an index basedon hierarchical associations between vertices of knowledge graphs andvertex adjacency in a knowledge graph, some embodiments may increase thespeed of document retrieval by using the index. Additionally, as furtherdescribed below, some embodiments may generate question-answer pairsbased on the knowledge graph and include the question-answer pairs in anindex. For example, some embodiments may associate a specific query ortype of query with a specific document or set of documents. Someembodiments may include this association representing a question-answerpair in the index.

FIG. 4 is a flowchart of an example of a process by which a query mayretrieve data based on a set of ontology graphs, in accordance with someembodiments of the present techniques. The process 400 may includeobtaining a query during a session, as indicated by block 404. A sessionmay include a login session between a client computing device and aserver or other computer system. During the session, one or more accountparameters of a user account of the session may be available. An accountparameter may include values such as a login identifier, username, asession identifier, an account identifier, a domain indicator, or thelike, where an account parameter space of an account parameter spacevector may include categorized or quantified values of accountparameters. For example, the query may include a natural language querysuch as “recent advances in health.”

In some embodiments, one or more account parameters may be computed froma set of stored activities. For example, some embodiments may determinea set of previously-accessed documents and determine a set of domainvectors based on the set of previously-accessed documents. Someembodiments may then determine a set of clusters of the set of domainvectors using a clustering method, such as a density-based clusteringmethod. For example, some embodiments may determine a count of domainvectors within a domain space region determined from dimensions of thedomain vector to select which set of domain vectors to in a cluster.Additionally, some embodiments may determine one or more accountparameters based on the set of clusters. For example, some embodimentsmay determine a first account parameter indicating a domain vectorrepresenting a centroid of the cluster. As discussed further below, someembodiments may use the text of the query or a set of account parametersto sort or otherwise filter a set of documents.

In some embodiments, the query may be generated as part of a decisionsupport system. For example, some embodiments may obtain inputsassociated with a decision to perform one or more operational changes.Some embodiments may generate a query based on a context of the system,input data, a role of a user account, or the like. For example, someembodiments may determine that a user assigned with a first user role istasked with determining whether to recommend an additional medical test.Some embodiments may retrieve a set of documents to provide guidancebased on an account of the decision-maker that includes an expertisescore for the domain associated with the decision. For example, someembodiments may determine that a user is associated with a “doctor” rolebased on an account corresponding to the user and is tasked with makinga decision on whether to schedule a first operation in the domain of“pulmonary health.” In response, some embodiments may provide guidancedocuments associated with the domain “pulmonary health” and associatedwith the class value corresponding to “doctor,” such as a review studyindicating statistical outcomes of the first operation. Additionally,some embodiments may determine that a second user is associated with a“nurse practitioner” role based on a second account corresponding to thesecond user and is tasked with making a decision on whether to schedulethe first operation. In response, some embodiments may provide guidancedocuments associated with the domain “pulmonary health” and associatedwith the class value corresponding to “nurse practitioner,” such as aguideline document instructing practitioners that the first operation isnot recommended with a second opinion. As described further below, someembodiments may use scores associated with a user account to determineappropriate hierarchy levels of an ontology graph or set of ontologygraphs and/or correspondingly appropriate documents.

The process 400 may include determining one or more learnedrepresentations based on the query, as indicated by block 408. Asdescribed elsewhere in this disclosure, a learned representation mayinclude a quantitative value, a category, a vector, a list of dataobjects, or the like. For example, a learned representation may includean embedding vector associated with an n-gram. Some embodiments may usethe same machine learning model as the ones described above. Forexample, some embodiments may use a trained encoder neural network oranother neural network to determine a set of vertices of an ontologygraph and use the same trained encoder neural network to determine thelearned representations of n-grams of the query. As further discussedbelow, some embodiments may expand a query using a hierarchical set ofontology graphs, where a learned representation may be linked to otherlearned representations using a cluster of vertices or other aggregationof learned representations in a domain space.

The process 400 may include determining a set of query scores based onthe set of learned representations or a set of parameters of thesession, as indicated by block 412. In some embodiments, a set of queryscores may be determined from a set of embedding vectors of a query. Forexample, the set of query scores may include a set of embedding vectors,additional values derived from the set of embedding vectors, a vectorrepresenting expertise scores in a set of domains, or the like.Alternatively, or in addition, some embodiments may determine the set ofquery scores based on a set of account parameters, where the set ofaccount parameters may include a login identifier, a hash value based onthe login identifier, data stored in an account of a user identified bythe login identifier, or the like. For example, some embodiments maydetermine a query score vector comprising a weighted sum of a firstdomain vector and a second domain vector, where the first domain vectormay include a set of domain indicators stored in a user account, andwhere the second domain vector may include a computed domain vectordetermined from the embedding vectors of the query.

The process 400 may include retrieving a set of stored documents basedon the query score and a set of ontology graphs, as indicated by block420. In some embodiments, a set of query scores for a query may becombined to form a query score vector. Some embodiments may use the setof query scores, either individually or in the form of a query scorevector, to determine which documents to retrieve based on one or moredocuments referenced by vertices or other elements of an ontology graph.For example, some embodiments may determine that a set of embeddingvectors of a query match with the first ontology graph's vertices, suchas an ontology graph of medical terminology. Some embodiments may expandthe query by determining associated concepts of the query via clustersor other aggregations of learned representations of n-grams of the queryin a domain space combining ontology graphs at different hierarchies.For example, some embodiments may receive a query and match an n-gram ofthe query to a first concept via an embedding vector of the n-gram beingpart of a cluster of vectors associated with the concept. A search toretrieve documents may result in documents that are indexed by theconcept, include the concept, or otherwise associated with the concept.Some embodiments may then retrieve a plurality of documents based on theset of embedding vectors of the query matching with one or more of theontology graph's embedding vectors. The retrieved plurality of documentsmay be obtained based on the documents referred to or otherwiseassociated with the ontology graph's vertices.

Some embodiments may use one or more machine learning models to retrievedocuments, summarizations based on documents, or the like as part ofproviding semantic search results after receiving a query. As discussedelsewhere in this disclosure, a machine learning model may include a setof decision trees forming a random decision forest, a neural networkhaving an attention mechanism, a neural network having one or morerecurrent neural network layers, a neural network having activationfunctions, ensemble models comprising different sub-models, or the like.For example, some embodiments may use a trained transformer neuralnetwork or other machine learning model to determine a set of dialogstates values for a query and use the dialog state values in conjunctionwith n-grams of the query or associated concepts of the n-grams toretrieve a document. Various dialog state values may be determined, andmay include an intent classification, a complexity classification, orthe like. Some embodiments may train an instance of a machine learningmodel using a first set of question-answer pairs, where machine learningparameters or hyperparameters may be transferred to other instances ofthe machine learning model. Some embodiments may implement suchparameter transfers as part of one or more transfer learning operations.

Various types of transfer learning operations may be performed. Forexample, some embodiments may use a set of transformer neural networksto select documents of a corpus for retrieval or processing based on then-grams of the document and metadata associated with the document. Usinga transformer neural network may converting n-grams into learnedrepresentations before determining one or more values for a dialog stateor other output of the transformer neural network. For example, atrained transformer neural network may determine a key value usable tosearch through an index of a set of n-grams of a document, an ontology,or a corpus, where the index may associate key values representingn-grams or ontology graph vertices with representations of other n-gramsor other ontology graph vertices. Some embodiments may perform one ormore latent feature learning operations on n-grams of a corpus or aninitial set of learned representations of the n-grams to determine alower dimensional set of learned representations.

As described in this disclosure, some embodiments may transferparameters of machine learning model, where the parameters may include aset of neural network parameters such as weights, biases, activationfunction parameters, or other values of the neurons of a neural network.Once transferred, these parameters may be used by a new instance of theneural network model or other machine learning model. For example, someembodiments can train a BERT-based machine learning model to predictanswers based on training queries from a stored library of queries andanswers, where the answers for the queries may include semantic searchresults. Some embodiments may train a machine learning model based on aset of training queries and a corresponding set of training documentsthat should be retrieved when the system is provided with the set oftraining queries. Additionally, some embodiments may substitute oraugment the stored library of questions and answers with a second set ofquestions and answers that are filtered by an associated domain or classwithin the domain. Some embodiments may perform inductive transferlearning operations, such as multi-task learning operations orsequential transfer learning operations. Performing a set of multi-tasktransfer learning operations may include concurrently training a machinelearning model (e.g., a recurrent neural network) to perform differenttasks. For example, some embodiments may perform multi-task transferlearning operations to by training a set of machine learning modelssharing one or more neural network layers to perform named entityrecognition, part-of-speech tagging, relationship extraction, or othertasks, where the operations may include one or more operations describedby Sanh et al (Sanh, V., Wolf, T. and Ruder, S., 2019, July. Ahierarchical multi-task approach for learning embeddings from semantictasks. In Proceedings of the AAAI Conference on Artificial Intelligence(Vol. 33, pp. 6949-6956)), which is incorporated herein by reference. Insome embodiments, the training data used to perform multi-task transferlearning operations or other training operations to train a machinelearning model described in this disclosure may include training thatuses questions as inputs and documents of the corpus, data based on orassociated with the documents of the corpus, or scores associated withthe documents of the corpus as outputs.

Some embodiments may perform a set of sequential transfer learningoperations by training a machine learning model using different sets oftraining data in a sequence. For example, some embodiments may train aninstance of a machine learning model with a first set of training dataand then train the pre-trained instance with a second set of trainingdata, where the second set of training data may be adapted to adomain-specific or user-specific set of purposes. Some embodiments maygenerate a pre-trained machine learning model with training data havinga set of training questions and set training documents from a corpus.Some embodiments may then adapt the pre-trained machine learning modelby its outputs for an additional set of layers of a neural network modelor another machine learning model (e.g., support vector machines, RandomForest, or the like). Alternatively, or in addition, some embodimentsmay use the transferred models as a starting set of parameters andfurther update the parameters based on additional training. For example,some embodiments may obtain an initial set of weights and biases forneurons of a neural network and update the set of weights and biases ofthe neural network during a second training operation with an additionalset of training operations using a second set of training data, wherethe second set of training data may be more domain-specific orclass-specific. Some embodiments may use an initial set of queries andexpected responses to the queries based on a corpus to train a machinelearning model using one or more operations described by Namazifar et al(Namazifar, M., Papangelis, A., Tur, G. and Hakkani-TUr, D., 2020.Language Model is All You Need: Natural Language Understanding asQuestion Answering. arXiv preprint arXiv:2011.03023), which isincorporated herein by reference. For example, some embodiments maytrain a machine learning model in a first stage based on apre-determined set of queries and responses, such as an ATIS datasetthat is augmented with a pre-determined set of questions and answersbased on the ATIS dataset. It should be recognized that some embodimentsmay use another dataset, such as an industry-specific or domainfield-specific dataset to perform the first stage of training with acorresponding set of pre-determined questions and answers. Someembodiments may then update the machine learning model by applying asecond training operation based on a specific class associated with thetraining data. For example, some embodiments may perform a secondtraining operation to generate a machine learning model that retrievestext from documents in response to a query based on a specific userclass of the user making the query.

Some embodiments may rank or filter the set of retrieved documents basedon one or more operations based on domain indicators or other valuesassociated with a query and domain indicators or other values associatedwith a user. For example, some embodiments may obtain a set of vectorsindicating one or more domain expertise scores of a user, where thevector may be obtained from a UI element or a corresponding useraccount. After providing a query, a user may be permitted to interactwith one or more UI elements to indicate their level of expertise in aset of domains, such as by selecting a category from a set of selectablecategories and writing a numeric value ranging between the numbers oneand five to indicate their preferred level of document complexity. Thisindicated preferred level of document complexity may then be used as adomain expertise score.

Some embodiments may retrieve a document by loading a document from arepository into a temporary or non-persistent memory. Alternatively,some embodiments may retrieve a document by loading an identifier, textportion, or other value based on the document into a temporary ornon-persistent memory. For example, some embodiments may retrieve adocument by retrieving the title of the document or another identifierof the document in a repository. A client computing device may thenobtain the text of the document by sending an additional request to acorresponding document-storing repository with the identifier of thedocument.

In some applications, only a single domain is considered when filteringor sorting a set of documents based on a domain expertise score. Someembodiments may sort a set of documents based on a hierarchy of a set ofontology graphs or their corresponding hierarchy of domain expertisevalues. For example, a first document may be associated most with afirst ontology graph that is itself associated with a domain score ofone, and a second document may be associated with a second ontologygraph that is itself associated with a domain score of eight. Afterobtaining a query associated with a domain score equal to seven, someembodiments may select the second document for retrieval and not selectthe first document for retrieval. Alternatively, after retrieving bothdocuments, some embodiments may display the second document at the topof a list of documents in a UI window, where the first document may beat a lower position in the UI window than the second document.

For example, if a user is associated with a domain class vector of “[0,5, 3],” representing expertise scores in three different domains, someembodiments may rank a set of documents based on their distance from thedomain vector in a domain class space, where the domain class vector maybe used as a query score vector. In some embodiments, the distance maybe used as a relevance score for a document and may indicate thelikelihood that the document will be considered meaningful or otherwiserelevant for a query provided by the user. Alternatively, or inaddition, the relevance score may be determined based on the distance.For example, some embodiments may determine a relevance score based onthe distance and based on the number of occurrences of n-grams sharedbetween the document and a corresponding query. After determining adistance measurement between the query score vector and each respectivedomain vector of a respective document of a set of documents or arelevance score based on the distance measurement, some embodiments maydetermine a ranking of the plurality of distance measurements and usethe ranking to determine which set of retrieved documents to display. Asdescribed above, a document domain vector may be determined based onword complexity, phrase complexity, syntax, grammar, or other featuresof a text document. Additionally, or alternatively, some embodiments mayupdate or generate a domain vector for a document based on the count andtypes of vertices of an ontology graph that corresponds with theembedding vectors of the document.

Some embodiments may then provide the set of retrieved documents to acomputing device for display, as indicated by block 430. someembodiments may obtain a vector indicating a domain expertise scorebased on data stored in association with a user account or other type ofaccount data or determined from a query score computed from the set ofembedding vectors of a query described above. For example, someembodiments may obtain an expertise score based on a set of user accountdata indicating that a user has an interest in “cardiology” and has anexpertise score of “9” in association with the domain “cardiology.” Inresponse, some embodiments may rank the set of retrieved documents basedon their distance to the set of domain expertise scores in a domainexpertise dimension space.

Providing the set of retrieved documents may include sending a list ofthe identifiers and corresponding text of the set of retrieved documentsto a client computing device. Alternatively, some embodiments mayinitially send the list of the identifiers of the documents in anordered sequence to a client computing device. In response to aselection of an identifier in a UI element being displayed on thecomputing device, some embodiments may then provide the text of theselected document. As discussed elsewhere in this discussion, someembodiments may use one more indices updated based on a set of ontologygraphs to reduce the time or computational resource use required toprovide a set of documents based on a query.

FIG. 5 shows an example of a computer system by which the presenttechniques may be implemented in accordance with some embodiments. FIG.5 is a diagram that illustrates an exemplary computer system 500 inaccordance with embodiments of the present technique. Various portionsof systems and methods described herein, may include or be executed onone or more computer systems similar to computer system 500. Further,processes and modules described herein may be executed by one or moreprocessing systems similar to that of computer system 500.

Computer system 500 may include one or more processors (e.g., processors510 a-510 n) coupled to system memory 520, an input/output I/O deviceinterface 530, and a network interface 540 via an input/output (I/O)interface 550. A processor may include a single processor or a pluralityof processors (e.g., distributed processors). A processor may be anysuitable processor capable of executing or otherwise performinginstructions. A processor may include a central processing unit (CPU)that carries out program instructions to perform the arithmetical,logical, and input/output operations of computer system 500. A processormay execute code (e.g., processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination thereof) thatcreates an execution environment for program instructions. A processormay include a programmable processor. A processor may include general orspecial purpose microprocessors. A processor may receive instructionsand data from a memory (e.g., system memory 520). Computer system 500may be a uni-processor system including one processor (e.g., processor510 a), or a multi-processor system including any number of suitableprocessors (e.g., 510 a-510 n). Multiple processors may be employed toprovide for parallel or sequential execution of one or more portions ofthe techniques described herein. Processes, such as logic flows,described herein may be performed by one or more programmable processorsexecuting one or more computer programs to perform functions byoperating on input data and generating corresponding output. Processesdescribed herein may be performed by, and apparatus can also beimplemented as, special purpose logic circuitry, e.g., an FPGA (fieldprogrammable gate array) or an ASIC (application specific integratedcircuit). Computer system 500 may include a plurality of computingdevices (e.g., distributed computer systems) to implement variousprocessing functions.

I/O device interface 530 may provide an interface for connection of oneor more I/O devices 560 to computer system 500. I/O devices may includedevices that receive input (e.g., from a user) or output information(e.g., to a user). I/O devices 560 may include, for example, graphicalUI presented on displays (e.g., a cathode ray tube (CRT) or liquidcrystal display (LCD) monitor), pointing devices (e.g., a computer mouseor trackball), keyboards, keypads, touchpads, scanning devices, voicerecognition devices, gesture recognition devices, printers, audiospeakers, microphones, cameras, or the like. I/O devices 560 may beconnected to computer system 500 through a wired or wireless connection.I/O devices 560 may be connected to computer system 500 from a remotelocation. I/O devices 560 located on remote computer system, forexample, may be connected to computer system 500 via a network andnetwork interface 540.

Network interface 540 may include a network adapter that provides forconnection of computer system 500 to a network. Network interface may540 may facilitate data exchange between computer system 500 and otherdevices connected to the network. Network interface 540 may supportwired or wireless communication. The network may include an electroniccommunication network, such as the Internet, a local area network (LAN),a wide area network (WAN), a cellular communications network, or thelike.

System memory 520 may be configured to store program instructions 524 ordata 515. Program instructions 524 may be executable by a processor(e.g., one or more of processors 510 a-510 n) to implement one or moreembodiments of the present techniques. Program instructions 524 mayinclude modules of computer program instructions for implementing one ormore techniques described herein with regard to various processingmodules. Program instructions may include a computer program (which incertain forms is known as a program, software, software application,script, or code). A computer program may be written in a programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages. A computer program may include a unit suitable foruse in a computing environment, including as a stand-alone program, amodule, a component, or a subroutine. A computer program may or may notcorrespond to a file in a file system. A program may be stored in aportion of a file that holds other programs or data (e.g., one or morescripts stored in a markup language document), in a single filededicated to the program in question, or in multiple coordinated files(e.g., files that store one or more modules, sub programs, or portionsof code). A computer program may be deployed to be executed on one ormore computer processors located locally at one site or distributedacross multiple remote sites and interconnected by a communicationnetwork.

System memory 520 may include a tangible program carrier having programinstructions stored thereon. A tangible program carrier may include anon-transitory computer readable storage medium. A non-transitorycomputer readable storage medium may include a machine-readable storagedevice, a machine readable storage substrate, a memory device, or anycombination thereof. Non-transitory computer readable storage medium mayinclude non-volatile memory (e.g., flash memory, ROM, PROM, EPROM,EEPROM memory), volatile memory (e.g., random access memory (RAM),static random access memory (SRAM), synchronous dynamic RAM (SDRAM)),bulk storage memory (e.g., CD-ROM or DVD-ROM, hard-drives), or the like.System memory 520 may include a non-transitory computer readable storagemedium that may have program instructions stored thereon that areexecutable by a computer processor (e.g., one or more of processors 510a-510 n) to cause the subject matter and the functional operationsdescribed herein. A memory (e.g., system memory 520) may include asingle memory device or a plurality of memory devices (e.g., distributedmemory devices). Instructions or other program code to provide thefunctionality described herein may be stored on a tangible,non-transitory computer readable media. In some cases, the entire set ofinstructions may be stored concurrently on the media, or in some cases,different parts of the instructions may be stored on the same media atdifferent times.

I/O interface 550 may be configured to coordinate I/O traffic betweenprocessors 510 a-510 n, system memory 520, network interface 540, I/Odevices 560, or other peripheral devices. I/O interface 550 may performprotocol, timing, or other data transformations to convert data signalsfrom one component (e.g., system memory 520) into a format suitable foruse by another component (e.g., processors 510 a-510 n). I/O interface550 may include support for devices attached through various types ofperipheral buses, such as a variant of the Peripheral ComponentInterconnect (PCI) bus standard or the Universal Serial Bus (USB)standard.

Embodiments of the techniques described herein may be implemented usinga single instance of computer system 500 or multiple computer systems500 configured to host different portions or instances of embodiments.Multiple computer systems 500 may provide for parallel or sequentialprocessing/execution of one or more portions of the techniques describedherein.

Those skilled in the art will appreciate that computer system 500 ismerely illustrative and is not intended to limit the scope of thetechniques described herein. Computer system 500 may include anycombination of devices or software that may perform or otherwise providefor the performance of the techniques described herein. For example,computer system 500 may include or be a combination of a cloud-computersystem, a data center, a server rack, a server, a virtual server, adesktop computer, a laptop computer, a tablet computer, a server device,a client device, a mobile telephone, a personal digital assistant (PDA),a mobile audio or video player, a game console, a vehicle-mountedcomputer, or a Global Positioning System (GPS), or the like. Computersystem 500 may also be connected to other devices that are notillustrated, or may operate as a stand-alone system. In addition, thefunctionality provided by the illustrated components may in someembodiments be combined in fewer components or distributed in additionalcomponents. Similarly, in some embodiments, the functionality of some ofthe illustrated components may not be provided or other additionalfunctionality may be available.

II. Cross-Class Ontology Integration

Workflows in various domains involve a mixture of general knowledgeunderstanding, domain-specific understanding, and quantitativeunderstanding. A significant challenge to using NLP for domain-specificworkflows is the gap between a general knowledge environment and adomain-specific environment. Domain-specific knowledge may includediffering interpretations of shared terminology, domain-specific logicalrelationships that differ between domains, patterns that are apparent orrelevant only in specific domains, quantitative relationships, or thelike. Additionally, domain-specific knowledge may be contradictory toknowledge encoded in data provided by or otherwise obtained from ageneral knowledge system. Thus, as a domain grows more specialized, NLPsystems trained on general knowledge systems may grow increasinglyunreliable when used within a specific domain. However, domain-specificdata is often considerably less voluminous relative to general knowledgedata, which may make domain-specific data inadequate for various NLPtraining tasks. The unreliability of NLP systems may then generate orexacerbate errors in a decision-support system, which may result in poorrecommendations or automated responses in domain-specific workflows.

As discussed elsewhere in this disclosure, some embodiments include aplurality of ontologies associated with different domains. In manyinstances, these ontologies may be independently using a set ofdifferent corpuses or implemented with different ontological objectives.Integrating these ontologies may often prove useful to acceleratecross-domain searches and determining new insights based on cross-domainknowledge. However, the integration of these domains may be madedifficult due to different logical relationships, domain vocabulary, orcategorical locations.

Some embodiments may integrate a first ontology with other ontologies toform the ontology system. Some embodiments may integrate the firstontology with other ontologies based on a second ontology associatingwords or other n-grams of the first ontology with the other ontologies,allowing words and concepts to be hierarchically linked across differentdomains or classes of expertise within the domains. Some embodiments mayinclude a set of UI elements to control a modular knowledge system andprovide visual indicators to indicate whether an ontology combinationpasses or fails a set of rules. Some embodiments may presentvisualizations of the different types of edges governing vertexrelationships within an ontology graph or other vertex relationships,where the ontology graph may represent an ontology. Some embodiments mayfurther present visualizations of query interpretations based on the setof ontology combinations. Some embodiments may include thesevisualizations in a decision-support platform. In some embodiments, thedecision-support platform may provide human users with visual indicatorsor UI elements to view or modify a set of parameters used by thedecision-support platform to provide recommendations or take actions.

By performing the operations described above, some embodiments mayconstruct a knowledge fabric usable for a decision-support platform inaccordance with an NLP system trained across multiple domain levels.Some embodiments may perform general language encoding ordomain-specific language encoding on text. Additionally, someembodiments may convert other forms of media (e.g., video, audio,images) into text data for analysis and incorporate the information inone or more domain levels. Some embodiments may perform query expansionusing the set of ontology graphs, a trained learning system, or thelike. Some embodiments may search through related knowledge systemsbased on the set of ontology graphs to provide additional graph-basedrelationships corresponding to domain-specific insights or cross-domaininsights in response to updates to the set of ontology graphs. Someembodiments may analyze and update elements of an ontology graph orother elements of a structured knowledge base in response to userfeedback to increase the speed and accuracy of a decision-supportsystem.

In some embodiments, an NLP system, NLU system, or ArtificialIntelligence (AI) system may be combined with ingested documents andother data to create a structured knowledge base of a knowledge fabricusable to provide data in response to queries. For example, someembodiments may obtain documents, tagged media files, data generatedfrom media files (e.g., transcripts of videos, recognized objects invideos, or the like). Some embodiments may then classify the documentsand other data into a set of domain-specific categories and generate orotherwise update a set of ontology graphs based on the provided data. Asfurther discussed in this disclosure, some embodiments may thendetermine relationships between the set of ontology graphs using one ormore machine learning operations of an NLP system to construct orotherwise update a set of ontology graphs that includes one or moreontology graphs, where each graph may be specific to a domain or classwithin the domain. The set of ontology graphs may be used as part of aknowledge fabric, and may be used by one or more NLP, NLU, or other AIsystems to provide data, recommendations, workflow instructions, orprogrammed operations.

Some embodiments may use a layered approach with a hierarchical toolchain (“Cognitive Tower”) that may improve performance over othergeneral-domain AI systems. Some embodiments may use transfer learningfrom language models enhanced by domain-specific, enterprise-specific,or workflow-specific contextual models. Additionally, some embodimentsmay customize a workflow pipeline for specific contexts, which mayimprove the accuracy of output recommendations or instructions. Someembodiments may modify one or more learning parameters based on a set ofsupervised and unsupervised learning operations performed in response tohuman interaction, which may further improve user efficiency and systemaccuracy.

FIG. 6 shows an example of different statement expansions based on aninitial statement, in accordance with some embodiments of the presenttechniques. The statement “Client Arm is going tender after current CVAoversight” shown in the box 610 is repeated three times in the boxes611-613. For example, some embodiments may obtain a first ontology graphin a electrical knowledge domain that separates the statement “ClientArm is going tender after current CVA oversight” into the items“client,” “arm,” “going to tender,” “current,” “CVA,” and “oversight.”Using the first ontology graph, some embodiments may convert the term“client” into “our client,” convert the term “arm” into “ARM holdings,”convert the word “current” into the phrase “Power,” convert the item“CVA” into the phrase “Central Volume Allocation,” and convert the item“oversight” into the phrase “a mistake.” Using these mappings, someembodiments may provide the phrase “our client ARM Holdings is going totender after a mistake in the Central Volume Allocation of Power,” whichis shown in the box 631 using the first ontology graph.

Some embodiments may obtain a second ontology graph in a medicalknowledge domain that separates the statement “Client Arm is goingtender after current CVA oversight” into the items, “client,” “arm,”“going to tender,” “after,” “current,” “CVA,” and “oversight.” Using thesecond ontology graph, some embodiments may convert the word “client”into “patient,” convert the word “after” into the phrase “due to,”convert the word “current” into the phrase “ongoing,” convert theacronym “CVA” into the word “stroke,” and convert the word “oversight”into the word “misdiagnosis.” Using these mappings, some embodiments mayprovide the phrase, “patient's arm has become sensitive to the touch dueto the ongoing misdiagnosis of a stroke,” which is shown in the box 632using the second ontology graph.

Some embodiments may obtain a third ontology graph in a businessknowledge domain that separates the statement “Client Arm is goingtender after current CVA oversight” into the items, “client,” “arm,”“going to tender,” “after current,” “CVA,” and “oversight.” Using thethird ontology graph, some embodiments may convert the phrase “clientarm” into the phrase “Part of the business dealing with clients,”convert the phrase “going to tender” into the phrase “up for sale,”convert the phrase “after current” into the phrase “after ongoing,”convert the acronym “CVA” into the phrase “credit valuation adjustment,”and convert the word “oversight” into the phrase “oversight process.”Using these mappings, some embodiments may provide the phrase, “part ofthe business dealing with clients is up for sale after ongoing anoversight process for the Credit Valuation Adjustment,” which is shownin the box 633 using the third ontology graph.

FIG. 7 shows an example of an initial query and an expanded query, inaccordance with some embodiments of the present techniques. The query701 includes a first n-gram 710, a second n-gram 720, and a third n-gram730. As shown in FIG. 7, the first n-gram 710 is the word “very,” thesecond n-gram 720 is the word, “720,” and the third n-gram 730 is theword “people.” It should be recognized that while each word in the query701 is an n-gram of the query 701, some embodiments may use syllables,phrases, characters, or some combination thereof as n-grams.

As discussed in this disclosure, each of the n-grams 710, 720, and 730may be associated with other n-grams using a set of ontology graphs. Thefirst n-gram 710 may be associated with a first set of alternativen-grams shown in the box 711. The second n-gram 720 may be associatedwith a second set of alternative n-grams shown in the box 721. The thirdn-gram 730 may be associated with a third set of alternative n-gramsshown in the box 731. As further discussed below, two or more of then-grams in the box 711, 721, or 731 may be associated with differentdomains or classes. For example, the n-gram “homo sapien” may beassociated with a first embedding vector that is indicated to be part ofa first domain labeled “biology” via an associated ontology graph vertexof an ontology graph categorized as being of the first domain.Additionally, the n-gram “clients” may be associated with a secondembedding vector that is indicated to be part of a second domain labeled“business” via an associated ontology graph vertex of an ontology graphcategorized as being of the second domain.

Some embodiments may generate, either in sequence or in parallel, a setof expanded queries 740. Each of the expanded queries 740 may beassociated with a score associated with a search context, a programenvironment context, parameters of an account used to perform thesearch, or the like. For example, some embodiments may receive the query701 from a first user logged in with a first user account and generate afirst set of scores for each of the expanded queries 740 based on one ormore domains or an expertise scores associated with the first account.Some embodiments may receive the query 701 from a second user logged inwith a second user account and generate a second set of scores for eachof the expanded queries 740 based on one or more domains or an expertisescores associated with the second account, where the first and secondset of scores are different from each other. For example, a first scorefor an expanded query “very dissatisfied clients” may be 15% for a firstuser account indicated as being included in a first domain labeled“psychology physician” and may be may 95% for a second user accountindicated as being included in a second domain labeled “customeroutreach staff.” Additionally, some embodiments may determine additionalscores based on domains labeled “severely depressed patients.” Based ona ranking determined from the scores, some embodiments may select one ormore expanded queries, such as the expanded query 750.

FIG. 8 shows the use of ontology graphs associated with differentclasses when determining an expanded query, in accordance with someembodiments of the present techniques. The query “BMS-56 D. in aFib ptswith VTE” includes a first n-gram 810, second n-gram 820, third n-gram830, fourth n-gram 840, and fifth n-gram 850, representing the terms“BMS-56,” “D.,” “aFib,” “pts,” and “VTE,” respectively. Some embodimentsmay navigate between ontology graphs of different domains andhierarchies within a domain to determine one or more alternative n-gramsor associated concepts. For example, some embodiments may determine thatthe first n-gram 810 is associated with a first related n-gram“Apixaban” shown in the box 811 based on a first edge of a firstontology graph associated with a first domain. In some embodiments, thefirst edge may associate a first vertex of the ontology graphrepresenting the n-gram “BMS-56” with a second vertex of the ontologygraph representing the n-gram “Apixaban.” Similarly, some embodimentsmay determine that the second n-gram 820 is associated with a secondrelated n-gram “Dose” shown in the box 821 based on a second edge of asecond ontology graph. In some embodiments, the second edge mayassociate a third vertex representing the n-gram “D.” with a fourthvertex of the second ontology graph representing the n-gram “Dose,”where both the third vertex and fourth vertex are vertices of the secondontology graph. Similarly, some embodiments may determine that thefourth n-gram 840 is associated with a fourth related n-gram “Patients”shown in the box 841 based on a third edge of the ontology graph. Insome embodiments, the third edge associates a fifth vertex representingthe n-gram “pts” with a sixth vertex of the second ontology graphrepresenting the n-gram “Patients.”

As shown in FIG. 8, some embodiments may retrieve multiple terms basedon a respective domain or class category within the domain. For example,some embodiments may first determine that the third n-gram 830 isassociated with a fifth related n-gram “Atrial Fibulation” shown in thebox 831 based on a fourth edge of the second ontology graph. In someembodiments, the fourth edge may associate a seventh vertex of thesecond ontology graph representing the n-gram “aFib” with an eighthvertex of the second ontology graph representing the n-gram “AtrialFibulation.” Additionally, in some embodiments, a first cross-graph edgemay associate the eighth vertex with a ninth vertex representing then-gram “Non-Valvular Atrial Fibulation” shown in a box 832, the ninthvertex being a vertex of a third ontology graph. Additionally, in someembodiments, a second cross-graph edge may associate the eighth vertexwith a tenth vertex representing the n-gram “Valvular Atrial Fibulation”also shown in the box 832, the tenth vertex also being a vertex of thethird ontology graph. Some embodiments may then determine that the tenthvertex is associated with an eleventh vertex representing the n-gram“NVAF” via a third cross-graph edge, the eleventh vertex being a vertexof the second ontology graph. In some embodiments, the second and thirdontology graphs may share a domain labeled “Domain 2,” but be associatedwith different hierarchies, where the second ontology graph isassociated with the “class 2” class, and where the third ontology graphis associated with the “class 3.”

As indicated above, after updating a set of ontology graphs by formingcross-graph edges relating vertices of different ontology graphs, someembodiments may provide previously-undetected associations betweendifferent vertices. For example, the first cross-graph edge associatingthe eighth vertex of the third ontology graph with the ninth vertex ofthe second ontology graph may be used to determine an associationbetween the n-gram “aFib” and the n-gram “Valvular Atrial Fibrillation.”Additionally, some embodiments may detect previously non-establishedlinks between vertices of a same ontology graph by using edgesassociating vertices of ontology graphs having different domains orclasses. For example, as shown in the association between the n-gram“Atrial Fibrillation” written in the box 831 and the n-gram “NVAF”written in the box 834, some embodiments may detect an associationbetween the n-gram “atrial fibulation” and the n-gram “NVAF,” where theassociation may be recorded in one or more of the vertices or otherwisestored in a database (e.g., as an ontological triple). This associationmay be used to include the n-gram “NVAF” when expanding a query havingthe n-gram “aFib,” generating a set of expanded queries, or otherwiseperforming searches to retrieve documents using query that includes then-gram “aFib” or “Atrial Fibulation.”

Similarly, some embodiments may first determine that the fifth n-gram850 is associated with a related n-gram “Acute Venous Thromboembolism”shown in the box 851 based on a fifth edge of the second ontology graph.In some embodiments, the fifth edge may associate a twelfth vertex ofthe second ontology graph representing the n-gram “VTE” with athirteenth vertex of the second ontology graph representing the n-gram“Acute Venous Thromoboembolism.” Additionally, in some embodiments, afirst cross-graph edge may associate the thirteenth vertex with afourteenth vertex representing the n-gram “Deep Vein Thrombosis” shownin the box 852, the fourteenth vertex being a vertex of the thirdontology graph. Additionally, in some embodiments, a second cross-graphedge may associate the fourteenth vertex with a fifteenth vertexrepresenting the n-gram “DVT” shown in the second ontology graph, thesecond vertex being a vertex of the second ontology graph.

FIG. 9 shows a representation of ontology graphs associated withdifferent classes of a hierarchical set of ontology graphs, inaccordance with some embodiments of the present techniques. FIG. 9displays a set of ontology graphs 900, the set of ontology graphsincluding a first ontology graph 910 having vertices 911-912, a secondontology graph 940 having vertices 941-951, and a third ontology graph970 having vertices 971-973. The first ontology graph 910 is categorizedwith a first domain labeled “domain 1” and further categorized with aclass category “Class 1.” The first ontology graph 910 includes a vertex911 and a vertex 912, where the vertex 911 is labeled with the n-gram“BMS-56,” and where the vertex 912 is labeled with the n-gram labeled“Apixaban.” Additionally, the first ontology graph 910 includes asubgraph of additional vertices represented by the box 914, where thevertex 912 is associated with the subgraph of additional verticesrepresented by the box 914 via the ontology graph edge 913. As discussedelsewhere in this disclosure, some embodiments may determine additionalquery expansions, update indices, or perform other actions based onvertices of the subgraph represented by the box 914 after adetermination that a query or search retrieves the vertex 912.

Various types of data structures stored in computer memory may be usedto represent an ontology graph, such as the first ontology graph 910,the second ontology graph 940, or the third ontology graph 970. Forexample, some embodiments may store each vertex as a record in a datatable, where each respective identifier of a respective vertex may serveas an index value usable to retrieve the respective vertex and itsassociated values. Alternatively, or in addition, a vertex may be storedas a plurality of values distributed across a plurality of databases,arrays, data objects, or the like. For example, some embodiments maystore a first array of values, a second array of pairs or triplets ofvalues, and a data table of records. Each value of the first array mayinclude an identifier to uniquely identify a vertex. Each pair ortriplet of values of the second array may include a pair of the uniqueidentifiers of the vertices indicating an ontology graph edge. Forexample, if the vertex 911 and the vertex 912 have the uniqueidentifiers “911” and “912,” respectively, the pair of values mayinclude “[911, 912],” which may represent or otherwise indicate anontology graph edge associating the vertices 911 and 912. The data tableof records may include additional values associated with the vertex,such as a label of the vertex, domain values associated with the vertex,hierarchy values associated with the vertex (e.g., an expertise score),ontology graph containing the vertex, or the like. For example, the datatable of records may include a first record indexed by the identifier“z3613c-1,” which may represent the vertex 911, where the first recordincludes an “n-gram” field that is filled with the value “BMS-56.”Similarly, the data table of records may include a second record indexedby the identifier “q3335c-1,” which may represent the vertex 912, wherethe first record includes an “n-gram” field that is filled with thevalue “Apixaban.”

The second ontology graph 940 is categorized with a first domain labeled“Domain 2” and further categorized with a class category “Class 2,”which may indicate the second ontology graph 940 is a graph in “Domain2” of the class “Class 2.” The vertex 941 is labeled with the text “D.”and associated with the vertex 942, which is labeled with the text“Dose,” where the vertex 942 may be further connected to a set of othervertices represented by a box 961. As discussed elsewhere in thisdisclosure, an index may be constructed or updated based on the ontologygraph edge associating the vertex 941 and the vertex 942. For example,some embodiments may detect the second n-gram 820 “D.” and, in response,retrieve the n-gram “Dose” shown in the box 821 based on the indexconstructed from the edge associating the vertex 941 with the vertex942. Some embodiments may retrieve the n-gram “Dose” by referring to anindex of a set of records indicating an association between the n-gram“D.” and the n-gram “Dose,” where the index may be constructed from orotherwise updated using the second ontology graph 940. Alternatively, orin addition, some embodiments may retrieve the n-gram “Dose” by using toa set of ontology graph edges, such as an array of pairs of identifiersas discussed above. For example, some embodiments may determine that ann-gram is a label for a vertex identified by a first identifier of apair of identifiers and, in response, retrieve a vertex or itsassociated values (e.g., an identifier, a label, an associated category,a set of associated scores, or the like) that is identified by the otheridentifier of the pair of values.

As further shown for the second ontology graph 940, the vertex 943 islabeled with the text “pts” and associated with the vertex 944, which islabeled with the text “patients.” As discussed elsewhere in thisdisclosure, some embodiments may detect the fourth n-gram 840 “pts” and,in response, retrieve the fourth n-gram “patients” shown in the box 841.Some embodiments may perform this retrieval based on the ontology graphedge associating the vertex 943 and 944 using one or more operationssimilar to those described for retrieving the n-gram “Dose” in responseto detecting the n-gram “D.” Similarly, as described above, someembodiments may detect the first n-gram 810 “BMS-56” and, in response,retrieve the n-gram “Apixaban” shown in the box 812 based on theontology graph edge associating the vertices 911 to 912.

As further shown for the second ontology graph 940 and the thirdontology graph 970, indirect associations between vertices may be usedto expand a query or the rank scores associated with the query, where anindirect association may be characterized by a set of edges from a firstvertex to a second vertex that includes at least two graph edges. Forexample, the third n-gram 830 includes the text “aFib” and may beassociated with the n-gram “Atrial Fibulation” shown in the box 831based on the association between the vertex 948 labeled “aFib” and thevertex 949 labeled “Atrial Fibulation” using one or more of theoperations described above. Some embodiments may then, based oncross-graph edges associating the vertex 949 with the vertices 971 and972, retrieve the n-grams “Non-Valvular Atrial Fibrillation” and“Valvular Atrial Fibrillation” shown in the box 832. In someembodiments, the cross-graph edges may be stored in a memory in a formatsimilar to or different from ontology graph edges. For example, across-graph edge may be an ontology graph edge that is stored in aseparate array of values, where each entry in the separate array ofvalues indicates a first vertex, a first ontology graph comprising thefirst vertex, a second vertex, and a second ontology graph comprisingthe second vertex.

As disclosed elsewhere, some embodiments may prioritize or otherwiseupdate a ranking of similar n-grams with respect to a first n-gram. Forexample, the ranking associated with the n-grams “Non-Valvular AtrialFibrillation” and “Valvular Atrial Fibrillation” as shown by thevertices 971 and 972 may be based on the class values of the ontologygraphs they are part of For example, some embodiments may receive aquery from a user having an account indicating that the user is a “class3” user for the domain “domain 2.” In response, some embodiments mayupdate a ranking of documents retrieved using the term “aFib” such thatdocuments retrieved using an expanded query having the n-gram“Non-Valvular Atrial Fibulation” shown in the box 832 is assigned agreater priority in a search ranking. For example, a first document andsecond document may have been retrieved based on the term “aFib” and asecond document may have been retrieved based on the n-gram“Non-Valvular Atrial Fibulation.” The first document may have beeninitially assigned a relevance score of “90” and the second document mayhave been initially assigned a relevance score of “90.” As furtherdiscussed in this disclosure, various operations may be performed tomodify a relevance score, such as by adding, subtracting, multiplying,applying an exponential term, or the like, where it should be understoodthat the prioritization of a document in a set of query results may becaused by either an increase or decrease of a relevance score associatedwith a document. For example, some embodiments may prioritize greaterrelevance scores over lesser relevance scores such that documents withgreater relevance scores are displayed before documents with a lowerrelevance score are displayed. Alternatively, or in addition, someembodiments may display a title or text from a higher-score documentabove a title or text of a lower-score document on a UI screen.Alternatively, some embodiments may prioritize lesser relevance scoresover greater relevance scores such that documents with lesser relevancescores are displayed before documents with a greater relevance score ordisplayed higher on a UI screen than a document with a greater relevancescore.

As described above, the n-gram “Non-Valvular Atrial Fibrillation” shownin the box 832 may be associated with the n-gram “NVAF” shown in the box834, where this association may be used to expand a query to use theterm “NVAF” when searching for documents. In some embodiments, theassociation between the two n-grams may be determined based on theassociation between the vertex 972 labeled “Non-Valvular AtrialFibrillation” and the vertex 945 labeled “NVAF” using one or more of theoperations described above. As described above, some embodiments maygenerate an expanded query that includes the n-gram “NVAF” in place ofor in addition to the n-gram “Non-Valvular Atrial Fibrillation.”Alternatively, or in addition, some embodiments may generate an expandedquery that includes the n-gram “NVAF” in place of or in addition torelated terms such as “aFib” based on the indirect association betweenedges connecting the vertex 948, the vertex 949, the vertex 972, and thevertex 945. For example, after receiving a query including the n-gram“aFib,” some embodiments may generate an expanded query that includesthe n-gram “DVT” for use in a semantic search. As further describedbelow, some embodiments may update the second ontology graph 940 or anindex based on the second ontology graph 940 in response to a detectionof the multi-edge association between vertices. For example, if no edgesassociated the vertex 948 with the vertex 945, some embodiments mayconstruct an edge associating the two vertices, such as by adding avertex identifier pair to an array of edges, updating a set of recordsrepresenting one or both vertices, or the like.

As described above, the fifth n-gram 850 includes the text “VTE” and maybe associated with the n-gram “Acute Venous Thromboembolism” shown inthe box 851 based on the association between the vertex 946 labeled“VTE” and the vertex 947 labeled “Acute Venous Thromboembolism” usingone or more of the operations described above. Some embodiments maythen, based on cross-graph edges associating the vertex 947 with thevertex 973 of the third ontology graph 970, retrieve the n-gram “DeepVein Thrombosis.” Furthermore, as described above, the n-gram “Deep VeinThrombosis” shown in the box 852 may be associated with the n-gram “DVT”shown in the box 854, where this association may be used to expand aquery to use the term “DVT” when searching for documents. In someembodiments, the association between these two n-grams may be determinedbased on the association between the vertex 973 labeled “Deep VeinThrombosis” and the vertex 950 labeled “DVT” using one or more of theoperations described above.

As described above, some embodiments may generate an expanded query thatincludes the n-gram “DVT” in place of or in addition to the n-grams“VTE,” “Acute Venous Thromboembolism,” or “Deep Vein Thrombosis” basedon the vertex connections formed by the edges connecting the vertex 946,vertex 947, the vertex 973, and the vertex 950. Some embodiments mayupdate the second ontology graph 940 or an index based on the secondontology graph 940 in response to a detection of the multi-edgeassociation between vertices. For example, if no edges associated thevertex 946 with the vertex 950, some embodiments may generate an edgeassociating the two vertices, such as by adding a vertex identifier pairto an array of edges, updating a set of records representing one or bothvertices, or the like. Future queries that include the n-gram “VTE” maymore quickly or efficiently provide results based on the n-gram “DVT” asa result of the newly-generated edge. For example, some embodiments mayprioritize results received from accessing an index constructed from anontology graph instead of accessing the ontology graph directly. Someembodiments may update the index in response to a newly-generated edgebetween two vertices by including an additional link associating then-gram “DVT” and “VTE” in the index before using the updated index toperform subsequent searches using the n-gram “VTE.” Furthermore, itshould be noted that while the set of ontology graphs are displayed withn-grams, other labels may be used. For example, the vertices of the setof ontology graphs 900 may include embedding vectors as identifiers ofthe vertices, where two or more vertices may be labeled with the samen-gram while having different embedding vectors.

FIG. 10 is a flowchart of an example process by which a query may beexpanded based on a set of ontology graphs, in accordance with someembodiments of the present techniques. Operations of the process 1000may begin at block 1004. In some embodiments, the process 1000 mayinclude obtaining a query during a session, as indicated by block 1004.Some embodiments may obtain a query using one or more operationsdescribed above for block 404. For example, some embodiments may obtaina query may by receiving a query from a user via a client computingdevice, where the user may be logged into an account during a datasession. For example, a user may be using a native application, a webapplication executing in the context of a web browser, or otherapplication that permits the user to log into a user account with ausername and a password. The user account may store or otherwise beassociated with a set of parameters, such as an indicated expertisescore or other value associated with a class of documents, a set ofdomains, or the like. The account may also store or include links to ahistory of retrieved documents, feedback messages or indicators from theuser indicating the relevance of documents, a set of previously-enteredqueries, age, ethnicity, geographic location, or the like. As describedfurther below, some embodiments may use account parameters to determinethe relevance of a set of retrieved documents or a set of expandedqueries generated from an initial query.

In some embodiments, the process 1000 may include determining a set ofn-grams based on the query, as indicated by block 1008. Some embodimentsmay determine a set of n-grams using one or more operations describedabove for block 408. For example, some embodiments may determine thateach word of the query may be used as an n-gram, where one or more ofthe words may be modified or deleted based on a set of filters thatremove stop words, lemmatizes words, stems words, or the like.Alternatively, or in addition, some embodiments may determine an n-gramas a phrase that includes multiple words, a syllable, a combination ofwords and punctuation, or the like. For example, some embodiments maydetermine that the phrase “valvular atrial fibrillation” is an n-gram.

Some embodiments may store past queries and their corresponding resultswhile receiving a new query. Some embodiments may then determine a querymatching score based on the n-grams of the past queries and the n-gramsthe new query. For example, some embodiments may determine that a firstquery and a second query are 90% identical with respect to a sharednumber of n-grams, where the documents retrieved using the first queryare still stored in a record of past searches. Some embodiments maydetermine whether the query matching score satisfies a query matchingthreshold and, if so, retrieve the list of previously-retrieveddocuments of the first query in response to receiving the second query.

In some embodiments, the process 1000 may include performing one or moreoperations described below for blocks 1016, 1020, 1024, 1030, 1038, or1042 for one or more respective n-grams of the set of n-grams determinedabove, as indicated by block 1012. Some embodiments may perform one ormore of the operations for each n-gram of the set of n-grams.Alternatively, some embodiments may perform one or more of theoperations for a subset of n-grams of the set of n-grams, where theoperations may be terminated before all of the n-grams are processedafter a determination is made that a terminal state or aprocess-terminating condition has been satisfied.

In some embodiments, the process 1000 may include determining a firstvertex of a first ontology graph based on the respective n-gram, asindicated for block 1016. Some embodiments may determine the firstontology graph based on an index constructed from the ontology graph, areference table indicating ontology graphs or vertices of ontologygraphs associated with the n-gram, a set of records representing thefirst ontology graph or part of the first ontology graph, or the like.For example, some embodiments may determine that an index constructedfrom a first ontology graph includes the respective n-gram, where therespective n-gram is linked to or otherwise associated with a vertexidentifier of the first ontology graph. As described above, someembodiments may determine a vertex of the ontology graph by firstdetermining a learned representation of an n-gram and then determining avertex associated with the learned representation.

Some embodiments may determine that an n-gram is mapped to multiplelearned representations and return a corresponding multiple number ofvertices for one or more ontology graphs. Some embodiments may assign anassociated context-matching score to the learned representations orcorresponding vertices indicating a likelihood of relevance using astatistical model or machine learning model. For example, someembodiments may use a machine learning model that includes a neuralnetwork that uses one or more parameters of a user account as an inputto determine a context-matching score for an n-gram that indicates apredicted relevance of a vector or other learned representation of then-gram. As discussed further below, some embodiments may construct aplurality of expanded queries, where each vector is used at least onceby at least one expanded query of the plurality of expanded queries.

In some embodiments, the process 1000 may include determining a set ofadjacent vertices or indirectly associated vertices of the first vertex,as indicated for block 1020. Some embodiments may determine the set ofadjacent vertices using an index, such as an index constructed from orotherwise based on an ontology graph. For example, an index may beupdated to include an association between an identifier of the firstvertex and an identifier of a second vertex, where the association maybe encoded as a set of connected index nodes of a B-tree structure. Insome embodiments, the encoded association of an index may represent anedge associating the first vertex and the second vertex in an ontologygraph. Some embodiments may determine a plurality of adjacent verticesof the first vertex, where one or more of the operations described inthis disclosure for adjacent vertices may be performed for each adjacentvertex of the plurality of adjacent vertices. Some embodiments mayfurther determine differences in class value between a first vertex orits corresponding first n-gram and a second vertex or its correspondingsecond n-gram based on the difference in class values stored in theindex. As further described below, using data stored in or otherwiseassociated with an index indicating class value may increase theefficiency and semantic accuracy of a semantic search based on a query.

Some embodiments may determine a set of indirectly associated verticesof a first vertex by crawling through the edges associated with thevertices, where the edges may include ontology graph edges of a sharedontology graph or ontology graph edges that cross ontology graphs andconnect vertices from different ontology graphs. For example, a firstvertex of a first ontology graph may be associated with a second vertexof the first ontology graph, and the second vertex may be associatedwith a third vertex of a second ontology graph via a cross-graph edge,where the third vertex may be indirectly associated with the firstvertex with an ontology graph edge distance equal to two. As furtherdescribed elsewhere in this disclosure, some embodiments may assign oneor more criteria to the graph edges it is permitted to use whendetermining a set of adjacent or indirectly associated vertices.

Some embodiments may determine a set of indirectly associated verticesof the first vertex based on a maximum ontology graph edge distance fromthe vertex. For example, some embodiments may obtain all the vertices ofa set of ontology graphs that are associated with a first vertex bythree or less ontology graph edges. Alternatively, or in addition, someembodiments may determine the set of indirectly-associated verticesusing a criteria based on one or more categories associated with theedges of the vertices. For example, some embodiments may determine a setof indirectly associated vertices of the first vertex based on theindirectly-associated vertices being labeled as either a subset of afirst concept associated with the first vertex or a lower class conceptof the first concept of a first ontology graph. In some embodiments, alesser class concept of a first concept may be a concept of a secondontology graph, the second ontology graph having a lesser class valuethan the first ontology graph.

In some embodiments, the process 1000 may include determining whetherthe set of adjacent vertices or indirectly associated vertices includevertices of an ontology graph having a different class or domain, asindicated by block 1030. As described above, the adjacent orindirectly-associated vertices of a first vertex may include one or morevertices of another ontology graph that is associated with the n-gram.For example, the first vertex may represent a first concept of a firstontology graph that is associated with a second concept of a secondontology graph, where class values of the respective ontology graphs maybe used to organize the concepts into a hierarchical set of concepts. Insome embodiments, the association between the first and second ontologygraphs may be available based on a cross-graph association between thefirst vertex or an adjacent vertex of the first vertex with one or morevertices of a second ontology graph. As disclosed elsewhere in thisdisclosure, the second ontology graph may differ from the first ontologygraph with respect to a domain or class of knowledge. If a determinationis made that the set of adjacent vertices or indirectly associatedvertices include vertices of an ontology graph having a different classor domain, some embodiments may proceed to operations described forblock 1038. Otherwise, operations may proceed to operations describedfor block 1042.

In some embodiments, the process 1000 may include updating a set ofscores associated with the n-grams of the adjacent orindirectly-associated vertices based the associated class or domainvalues, as indicated by block 1038. For example, some embodiments maydetermine a set of n-gram weights associated with each n-gram based on ashared domain or class with respect to a user account. Using theconvention that a greater n-gram weight results in a greaterprioritization of the corresponding n-gram for use in generating anexpanded query, as further described below, some embodiments mayincrease the n-gram weight of an n-gram of an adjacent or indirectlyassociated vertex. For example, some embodiments may increase an n-gramweight for an n-gram based on the n-gram being associated with a vertexsharing a class value with a class value indicated by a user account.Alternatively, or in addition, some embodiments may increase or decreasean n-gram score based the number of ontology graph edges between a firstn-gram and a second n-gram. Furthermore, some embodiments may reduce thecost of an n-gram weight for a second n-gram with respect to a firstn-gram based on one or more values stored in an index associating thesecond n-gram with the first n-gram. It should be understood that someembodiments may instead rank weights of an n-gram such that a lesserweight results in a greater prioritization and reduce a weight insteadof increasing the weight to increase the prioritization of acorresponding n-gram.

In some embodiments, the process 1000 may include determining whether anadditional n-gram of the query should be processed using one or more ofthe operations described above, as indicated by block 1042. Someembodiments may process each of the n-grams of a query for example, someembodiments may obtain the initial query “what do babies eat,” use eachof the words of the initial query as an n-gram of the query, and performone or more of the operations described above for each of the n-grams“what,” “do,” “babies,” and “eat.” Alternatively, or in addition, someembodiments may process a subset of the n-grams of a query. For example,some embodiments may obtain a query “a very depressing people” and applya set of filtering operations such as lemmatizing, stopword removal, andstemming to produce a filtered query having the n-grams “very,”“depress,” and “people.” Some embodiments may then determine that then-gram “very” is a low-priority n-gram or is not part of any ontologygraphs that the user has permission to access and, in response, use then-grams “depress” and “people” as part of a query.

Some embodiments may generate a set of expanded queries based on the setof vertices described above, as indicated by block 1050. An expandedquery of an initial query may include n-grams from the initial query andn-grams associated with one or more of the vertices described above. Forexample, after receiving a first expanded query, “BMS-56 D. in aFib ptswith VTE,” some embodiments may generate a first expanded query thatincludes the n-grams “BMS-56,” “D.,” “aFib,” and “pts.” the firstexpanded query may also include the n-gram “DVT,” where the n-gram“acute venous thromboembolism” may be associated with a vertex that isadjacent to a vertex representing the n-gram “VTE.”

Some embodiments may prioritize use of n-grams having greater n-gramweights. For example, if the n-grams “Acute Venous Thrombosis” and the“DVT” have n-gram weights of 30 and 70 with respect to the n-gram “VTE,”some embodiments may prioritize the generation of an expanded queryusing the n-gram “DVT” over the generation of an expanded query usingthe n-gram “Acute Venous Thrombosis.” Alternatively, or in addition,some embodiments may rank a first query or semantic search results ofthe first query with a lesser score than a second query or semanticsearch results of the second query in response to the first query usinga n-gram having a lesser n-gram weight.

Some embodiments may use one or more machine learning operations togenerate one or more expanded queries. For example, as describedelsewhere in this disclosure, some embodiments may use an abstractivetext summarization model or other natural language processing model togenerate an expanded query based on the model. Some embodiments may usea pre-trained neural network, such as a neural network of a generativepre-trained transformer (GPT) language model or a neural network of abi-directional encoder-decoder model, to generate an expanded query,where the neural network may use a subset of the n-grams of aninitially-obtained query. For example, some embodiments may use atransformer neural network to determine a set of embedding vectors for aset of n-grams of a query using a set of encoder neural network layersof the transformer neural network. As described elsewhere in thisdisclosure, in some embodiments, the encoder neural network layers mayhave three or less neural network layers. Some embodiments may thendetermine a set of positional encoding vectors, where each positionalencoding vector may be determined based on a position of a respectiven-gram in the selected set of n-grams. Some embodiments may thengenerate a plurality of random feature maps based on the set ofembedding vectors using one or more feature map functions. For example,some embodiments may use a feature map function based on the set ofembedding vectors comprises generating a set of random or pseudorandomvariables and multiplying at least one variable of the set of random orpseudorandom variables with the at least one element of the set ofembedding vectors.

As described elsewhere in this disclosure, some embodiments may use atransformer neural network model that includes one or more attentionmechanisms to generate a query or other text. For example, someembodiments may use a transformer neural network that includesdetermining a set of attention values and using the attention values togenerate or update a query. After receiving a user-provided query, someembodiments may determine a set of embedding vectors based on then-grams of the user-provided query using the transformer neural network.For example, some embodiments may determine embedding vectors for then-grams of the query using an encoder neural network having three orfewer neural network layers, where having a lesser number of neuralnetwork layers may reduce the computation required to generate text.Some embodiments may then generate a first random feature map based onthe set of embedding vectors using a feature map function. In someembodiments, using the feature map function may include generating afirst set of random or pseudorandom variables and multiplying at leastone variable of the first set of random or pseudorandom variables withthe at least one element of the set of embedding vectors.

Some embodiments may then determine a set of positional encoding vectorsthat indicate a position of an n-gram with respect other n-grams and usethe positional encoding vectors as additional inputs of a neuralnetwork. For example, some embodiments may generate a second randomfeature map based on the set of positional encoding vectors usinganother feature map function, where using the random feature mapincludes multiplying at least one variable of a set of random orpseudorandom variables with the at least one element of the set ofpositional encoding vectors. Some embodiments may then determine a setof attention values based on the first random feature map and the secondrandom feature map, such as by performing a set of element-wise matrixoperations. Some embodiments may then generate an expanded query usingthe neural network based on the set of attention values. For example,some embodiments may use a neural network having neural network layersthat use one or more of the set of attention values as inputs todetermine additional n-grams for an expanded query or to determine newn-grams for use as substitute n-grams for n-grams of a user-providedquery.

Some embodiments may determine the set of documents or set of associatedscores based on the set of expanded queries as indicated by block 1054.Some embodiments may perform one or more of the operations describedabove for block 420 to retrieve a set of documents or set of associatedscores based on a query. For example, some embodiments may obtain anindex constructed from or otherwise updated with data from one or moreof the ontology graphs described above. The index may include a set oflists, arrays, or other elements that link or otherwise associaten-grams to documents. For example, the index may include an arrayassociating a first and second identifier.

As described elsewhere in this disclosure, some embodiments maydetermine a score for a document based on data stored in the documentand a set of account parameters. For example, if a user having anaccount indicating a class value provides a query, some embodiments mayincrease a relevance score of a retrieved document if the retrieveddocument is also labeled with the same class value. Some embodiments mayupdate parameters of a scoring function used to determine a relevancescore. For example, after displaying a plurality of expanded queriesthat includes a first query having a greatest relevance score and asecond query having a relevance score less than that of the first query,some embodiments may receive feedback indicating that an expanded queryis a preferred query. In response, some embodiments may update an n-gramweight associated with a third n-gram of the second expanded query,where the n-gram weight may be a parameter of a scoring function used togenerate or rank at least one query of the plurality of expandedqueries, and where the first expanded query does not include the thirdn-gram. By updating the n-gram weight or some other parameter used togenerate the plurality of expanded queries, some embodiments mayincrease the accuracy of an expanded query or its corresponding searchresult(s).

FIG. 11 is a flowchart of an example process by which a hierarchical setof ontologies may be updated, in accordance with some embodiments of thepresent techniques. Operations of the process 1100 may begin at block1110. In some embodiments, the process 1100 may include obtaining a setof ontology graphs, as indicated by block 1110. In some embodiments, theset of ontology graphs may be constructed using operations similar to orthe same as those described above. For example, some embodiments mayobtain a first ontology graph categorized as being part of a firstdomain and a first class value, a second ontology graph categorized asbeing part of the first domain and a second class value, and a thirdontology graph categorized as being part of a second domain and a thirdclass value.

In some embodiments, the process 1100 may include obtaining an updatefor the set of ontology graphs, as indicated by block 1120. In someembodiments, an update may be obtained from a computing device executingone or more operations described in this disclosure. Alternatively, orin addition, an update may be obtained from a third-party computingsystem and received at an application program interface of a serverexecuting one or more operations described in this disclosure.

Some embodiments may obtain an update from an interaction with a UI. Forexample, some embodiments may send program code to a native application,web browser, or other application executing on a computing device thatcauses a display screen to show a UI. The UI may include interactiveelements that allows a user to form connection lines or other connectingshapes between visualizations that represent concepts or other verticesof an ontology graph. After determining that the interaction with the UIelement connects a first and second concept (or two other vertices of anontology), some embodiments may send a message from the computing deviceindicating the association the first concept and the second concept.Additionally, in some embodiments, the UI may include program code thatstores a set of rules or other conditions.

In some embodiments, after determining that an interaction with a UIwould update a hierarchical set of graphs, some embodiments may verifywhether one or more of the set of rules or other conditions would beviolated. Various conditions may be applied and tested, such as acondition that restrict vertices of a first type from being associatedwith vertices of a second type, a condition that restricts n-gramsassociated with a first concept from being associated with a secondconcept, a condition that restricts vertices associated with a firstclass value from being associated with vertices having a different classvalue without an appropriate user authorization, or the like. Forexample, some embodiments may include a condition that a user logged invia a user account must have an appropriate permission value beforebeing permitted to edit a connection between a first vertex representinga first concept and a second vertex representing a second concept. Inresponse to a determination that a rule would be violated by a proposedconnection between vertices, a verification element of the UI may changetext or appearance (e.g., change a color, shape, size, or the like) toindicate that the rule would be violated by the proposed connectionother proposed update to a set of ontology graphs.

In some embodiments, the UI may include additional UI elements to updateother operations described in this disclosure. For example, aninteraction with a UI element may re-arrange blocks representingworkflow operations such as document ingestion, learned representationgeneration, other NLP operations, other machine learning operations,ontology modification, or the like. Some embodiments may provide a UIthat permits a user to update a workflow block representing one or moreworkflow operations to indicate a machine learning model, parameters ofthe machine learning model, a set of ontology graphs to update, or thelike. For example, some embodiments may provide a UI that permits a userto add workflow blocks to a set of workflow blocks, remove workflowblocks from the set of workflow blocks, reconfigure workflow blocks ofthe set of workflow blocks, or otherwise update the set of workflowblocks. In response to a change to the set of workflow blocks, someembodiments may update a compiled version of a program code implementinga set of operations represented by the set of the workflow blocks.

In some embodiments, the process 1100 may include updating a set ofontology graphs based on the update, as indicated by block 1130.Updating a set of ontology graphs may include updating edges connectingvertices of the same or different ontology graphs, updating n-grams orword embeddings associated with the edges, updating documents associatedwith the vertices, or the like. In some embodiments, new ontology graphedges may be constructed to associated different vertices of an ontologygraph based on associations with vertices of a second ontology graph,where the associations with the second ontology graph may be caused byan update message. For example, a first ontology graph associated with afirst domain and class may include a first vertex mapped to a firstn-gram. The first vertex may be associated with a first embedding vectorof the first n-gram, where the first embedding vector is a vector of afirst cluster. The first cluster may represent a first concept that ismapped to a second n-gram and a corresponding second vertex, where thesecond n-gram may represent a centroid of the first cluster or amanually-entered label for the first cluster. Similarly, a secondontology graph associated with a second domain or class may include athird vertex mapped to a third n-gram. The third vertex may beassociated with a second embedding vector of the third n-gram, where thesecond embedding vector is a vector of a second cluster. The secondcluster may represent a second concept that is mapped to a fourth n-gramand a corresponding fourth vertex, where the fourth n-gram may representa centroid of the second cluster or a manually-entered label for thesecond cluster.

Some embodiments may obtain instructions to associate the first conceptand the second concept. For example, some embodiments may associate apair of concepts to each other based on a shared set of n-grams, ashared set of documents, a user-entered association, or the like. Theconcepts may be associated to each other via an association betweenn-grams representing the concepts, an association between embeddingvectors representing the concepts, an association between vertices of aset of ontology graphs representing the concepts, or the like. Inresponse to the association between the concepts, some embodiments mayconstruct an edge between the first n-gram and the third n-gram based ona first edge associating the first n-gram with the second n-gram, asecond edge associating the second n-gram with the fourth n-gram, and athird edge associating the fourth n-gram with the fourth n-gram.

Some embodiments may receive an update comprising instructions toassociate the first cluster and the second cluster based on a shared setof n-grams, a shared set of documents, a user-entered association, orthe like. In response, some embodiments may generate an ontologicalgraph edge between the first n-gram and the third n-gram based on afirst edge associating the first n-gram with the second n-gram, a secondedge associating the second n-gram with the fourth n-gram, and a thirdedge associating the fourth n-gram with the fourth n-gram. Someembodiments may generate the ontological graph edge by generating anontological triple comprising identifiers for a vertex representing thefirst n-gram and the second n-gram and storing the ontological triple ina database of ontological triples.

In some embodiments, obtaining the update may include obtaining arequest to associate a first element of a set of ontology graphs withanother element of the set of ontology graphs, where the element mayinclude a vertex of an ontology graph, a concept, an n-gram associatedthe vertex, or the like. For example, a user may type in a data entryindicating that the n-gram “heart attack” is associated with the concept“cardiovascular emergency.” In some embodiments, the update may beassociated with a given domain based on a domain or other domaincategory value assigned to a user providing the update. For example, theupdate may be associated with the domain of knowledge “neurology” basedon a determination that the update provider's associated domains includethe domain “neurology.”

Some embodiments may select one or more associated ontologies to whichthe update is applicable based on one or more domain category valuesassociated with the update. For example, after receiving an updaterequest associated with a first domain of knowledge associated with afirst ontology graph, class value within the first domain of knowledge,or other domain category value, some embodiments may select a secondontology graph from amongst a plurality of ontology graphs as also beingan applicable ontology graph with respect to the update request. Theselection of the second ontology graph may be based on a determinationthat the first and second ontology graphs are related based on thedomain category value. For example, some embodiments may select thesecond ontology graph based on a determination that the first and secondontology graphs share a domain category value, such as both first andsecond ontology graphs sharing the domain category “medicine” anddiffering with respect to their corresponding class values.

Alternatively, or in addition, some embodiments may select one or moreontology graphs based on their respective domain category distance(s). Adomain category distance may include a distance (e.g., Manhattandistance, Euclidean distance, or the like) in a domain category valuespace. For example, if a domain category distance is calculated using aManhattan distance, the domain category distance between a firstontology graph and a second ontology graph may be equal to thedifference between their respective class values. Some embodiments maythen determine whether two ontologies are associated with each otherbased on whether the domain category distance satisfies a distancethreshold. Alternatively, or in addition, some embodiments may determinethe domain category distance for an ontology graph based on differencesbetween a domain category value (e.g., a class value) of the ontologygraph and an account parameter of the user account used to provide anupdate. In some embodiments, if the domain category distance satisfies adistance threshold, the corresponding ontology graph may be selected,and if the domain category distance does not satisfy the distancethreshold, the corresponding ontology graph may not be selected.

Alternatively, or in addition, some embodiments may select an ontologygraph based on whether or not a provider of the update is associatedwith the ontology graph via the user account of the update provider. Forexample, some embodiments may select a second ontology graph based on adetermination that an identifier for the second ontology graph is anelement of an array stored in a user account of the update provider. Insome embodiments, the array may indicate the domains of knowledge thatthe provider is indicated to have expertise in, may indicate classvalues for one or more domains of knowledge, or may include other domaincategory values. In some embodiments, satisfying a domain threshold forupdating an ontology graph may include satisfying a requirement that theuser account lists the identifier of the ontology graph. Alternatively,or in addition, satisfying the domain threshold may include satisfying arequirement that the user account lists a quantitative score (e.g., anexpertise score) for the corresponding domain that satisfies aquantitative threshold (e.g., greater than one).

Some embodiments may search through a set of selected ontology graphsfor concepts or other vertices related to the first vertex. For example,after receiving an update indicating an association between an n-gramand a first concept, some embodiments may determine a first vertexassociated with the n-gram by generating an embedding vector based onthe n-gram and determining a first vertex mapped to the embeddingvector. Some embodiments may then determine whether the first concept isassociated with a second concept via a cross-graph edge that isindicated to permit an association between n-gram and the secondconcept.

Some embodiments may determine that an n-gram (or a vertex representingthe n-gram) may be associated with a concept of a different ontologygraph in response to their associated edges or the vertices between themsatisfying one or more criteria based on one or more relationship types.As discussed elsewhere in this disclosure, a relationship type may berepresented by a category or combination of categories associated with agraph edge(s) between two or more vertices and may indicate variousrelationships between n-grams or other elements represented by vertices.For example, a relationship type may indicate that a first n-gram is asubclass of a concept, is a symptom of a concept, is a cause of aconcept, is equivalent to a concept, or the like. Some embodiments maythen determine that two vertices or values they represent (e.g.,n-grams, concepts, or the like) may be associated based on adetermination that the edges between them are of the same type. Forexample, some embodiments may determine that a first vertex representingan update-provided n-gram is associated via first graph edge to a secondvertex representing the first concept, where the first graph edgeindicates that the first vertex is a subtype of the second vertex.Additionally, the second vertex may be associated with a third vertexrepresenting a second concept, where the third vertex is a vertex of asecond ontology graph, and where the second graph edge associating thesecond and third vertices indicate that the second graph edge indicatesthat the second vertex is a subtype of the third vertex. In response,some embodiments may determine that the first vertex and third vertexmay be associated, where such association may be performed byassociating one or more values of the first vertex with one or morevalues of the third vertex. For example, some embodiments may associatean n-gram represented by the first vertex with a concept of the thirdvertex.

Once a determination is made that a relationship criterion is satisfiedand that a first n-gram mapped to a first vertex of a first ontologygraph may be associated with a concept of a different ontology graph,some embodiments may then associate the first n-gram with an n-gram ofthe concept. For example, if the concept is directly mapped to a secondvertex or second n-gram of the second vertex, some embodiments may thenassociate the first n-gram with the second n-gram representing theconcept. Alternatively, or in addition, a third n-gram may be associatedwith a fourth vertex that is associated with the second vertex, and someembodiments may then associate the first n-gram with the third n-gram.

Some embodiments may update a set of indices based on updates to the setof ontology graphs, as indicated by block 1140. Operations to update anindex may include one or more operations described above for block 340.For example, some embodiments may update an index to associate a firstn-gram directly with a second n-gram in the index. Some embodiments mayupdate an index that is structured in the form of a B-tree, where a keyvalue corresponding to the first n-gram is stored in a parent node ofthe index and may be associated with a second n-gram via a leaf node ofthe parent node. Alternatively, or in addition, some embodiments mayupdate an index to associate a first n-gram with a document associatedwith the second n-gram.

Those skilled in the art will also appreciate that while various itemsare illustrated as being stored in memory or on storage while beingused, these items or portions of them may be transferred between memoryand other storage devices for purposes of memory management and dataintegrity. Alternatively, in other embodiments some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated computer system via inter-computercommunication. Some or all of the system components or data structuresmay also be stored (e.g., as instructions or structured data) on acomputer-accessible medium or a portable article to be read by anappropriate drive, various examples of which are described above. Insome embodiments, instructions stored on a computer-accessible mediumseparate from computer system 500 may be transmitted to computer system500 via transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as a network or a wireless link. Various embodiments may furtherinclude receiving, sending, or storing instructions or data implementedin accordance with the foregoing description upon a computer-accessiblemedium. Accordingly, the present techniques may be practiced with othercomputer system configurations.

FIG. 12 is a logical architecture indicating the integration of a datasystem with one or more learning systems, in accordance with someembodiments of the present techniques. The logical architecture 1200includes a learning model repository 1202 that may be accessed toprovide an initial pre-trained head 1212 of a learning model 1210, wherethe pre-trained head 1212 may include a first set of neural networklayers, a first set of ontology graphs, model hyperparameters, or thelike. The learning model repository 1202 may include parameters andfunctions corresponding to one or more types of learning models, such asBERT, BioBERT, GPT-3, USE, RoBERTa, ELMo, T5, EXLNet, BART, or the like.For example, the learning model repository 1202 may include scripts orprogram code usable for executing instances of a learning model and aset of neural network parameters such as weights, biases, activationthresholds, activation function parameters, or the like.

Some embodiments may perform transfer learning operations to provide thepre-trained head 1212 of the learning model 1210. Some embodiments maythen use the initial set of parameters of the pre-trained head 1212 togenerate an initial output that is usable as an input for a set oftask-specific layers 1214 during the training of the learning model1210, where some embodiments may use hierarchical ontologies to generatethe initial output. The initial output may include categories or a setof embedding vectors corresponding to training inputs that are then usedby the set of task-specific layers 1214.

The set of task-specific layers 1214 may include a second set of neuralnetwork parameters that are updated with a set of training operations toperform various tasks. In some embodiments, the set of trainingoperations may use parameters obtained from a template task library 1204to update parameters of the set of task-specific layers 1214. Thetemplate task library 1204 may include training task parameters such asa training dataset for tasks such as summarization, text classification,language modeling, named entry recognition, text encoding, ontologylookup, natural language generation, question-answering, textrepresentation, or the like. For example, some embodiments may train thelearning model 1210 by importing neural network parameters from thetemplate task library 1204 into a set of neural network layers of theset of task-specific layers 1214 and use a first training dataset of thetemplate task library 1204 to train the modified learning model 1210.

In some embodiments, the learning model 1210 may include or otherwiseaccess a set of ontology graphs, such as ontology graphs obtained fromthe learning model repository 1202 or a data store used store ontologygraphs. For example, the learning model 1210 may access a graph databaseto import a set of ontology graphs corresponding with different domainsor domain levels, such as those described in this disclosure. Someembodiments may use these ontology graphs to perform one or more tasksdescribed in this disclosure. For example, some embodiments may performnamed entity recognition to recognize a set of words as a named entity,assign a first category value of a first ontology graph to the namedentity, and assign a second category value of a second ontology graph tothe same named entity.

After generating or updating the learning model 1210, some embodimentsmay perform a set of fine-tuning operations represented by the fine tunetraining function 1220, where the fine tune training function 1220 mayapply data from dataset(s) 1206. The dataset(s) 1206 may includepublicly available “open” datasets, data specific to an account ororganization, additional annotations on a document (e.g., user-enteredclassifications or named entities), or the like. where the fine tunetraining function 1220 may be limited to updating a subset of theparameters of the learning model 1210. For example, some embodiments mayupdate the set of task-specific layers 1214 with a first dataset of thetemplate task library 1204 by updating parameters of three neuralnetwork layers of the set of task-specific layers 1214 and then updatingparameters of only two neural networks of the set of task-specificlayers 1214 with the fine tune training function based on the dataset(s)1206. Alternatively, some embodiments use additional training operationsbased on the dataset(s) 1206 to update some or all of the parameters ofthe set of task-specific layers 1214 or some or all of the parameters ofthe learning model 1210. For example, some embodiments may update a setof ontology graphs to indicate new associations between differentconcepts encoded in the set of ontology graphs, add new verticesrepresenting a new concept with its corresponding n-gram, or otherwiseupdate the set of ontology graphs. Some embodiments may store parametersof the learning model 1210 in the trained model storage 1230 afterperforming one or more of the training operations described above.

Some embodiments may access the trained model storage 1230 whenproviding artificial intelligence (AI) services for tasks described inthe utility set 1240. In some embodiments, the AI services may includestateless API services that provide access to one or more trained modelsdescribed in this disclosure. The operations of the utility set 1240 mayinclude a content intelligence operation 1242, a decision treemanagement operation 1246, a document tracking and comparison operation1244, a document processing operation 1248, a parameter-trackingoperation, or other operations described in this disclosure. Forexample, some embodiments may use the model and model parameters of thetrained model storage 1230 to generate summarizations of texts, generatequeries for indexing operations, or the like, as described elsewhere inthis disclosure.

Some embodiments may present the results of the AI services 1234 to a UIthat includes UI elements enabling a user to provide feedback. Forexample, some embodiments may display a generated text to a user in aUI. The user may click on a word of the UI and select one or moredomains or domain level that should be assigned to the word, where theselection of the word, domains, or domain levels may be sent in amessage to a feedback system 1250. The feedback system 1250 may thenupdate the dataset(s) 1206 to include the user-updated assignment of theword to the domains or domain levels, which may also cause the fine tunetraining function 1220 to update the parameters of the learning model1210. For example, some embodiments may update a set of named entityrecognition operations based on the updated feedback, where the set ofnamed entity recognition operations may be used in a document comparisonoperation.

As discussed elsewhere in this disclosure, some embodiments may detectchanges in concepts or other categories assigned to a word, name,phrase, or other n-grams across different documents or differentversions of a document. For example, some embodiments may determine thatthe concept represented by the n-gram “burger” is associated with afirst set of other concepts including the concepts represented by then-grams “sandwich” and “lunch” based on the text of a first set ofdocuments. Some embodiments may then determine that the conceptrepresented by the n-gram “burger” is associated with another conceptrepresented by the n-gram “vegan” based on a second set of documents,where the second set of documents may have been authored or obtained ata different time than the first set of documents and update thecorresponding ontology graph(s) to indicate the association between theconcept represented by “burger” and the concept represented by “vegan.”

Various criteria may be used to determine associations between conceptsor other elements representable by n-grams. For example, after firstrecognizing that two different n-grams represent two different namedentities, some embodiments may determine that the two different namedentities are associated with each other based on a determination thatthe frequency by which two corresponding n-grams are in the samesentence together across multiple documents is greater than a frequencythreshold. Alternatively, or in addition, some embodiments may determinethat the two named entities are associated with each other based on adetermination that the embedding vectors corresponding to the pair ofnamed entities are sufficiently close to each other in the embeddingspace of the embedding vectors. Some embodiments may further useadditional words or corresponding embedding vectors to determine ahierarchical relationship between the two named entities or may use theontology graphs themselves to determine the hierarchical relationship.For example, some embodiments may determine that the n-gram “veggieburger” is a subset of the n-gram “burger” in an ontology graph based onprevious graph edges of the n-gram “veggie burger” and the n-gram“burger.”

Some embodiments may update a set of ontology graphs to indicate that afirst n-gram is associated with a second n-gram in either the samedomain category or across different domain categories. For example, inan initial generation of a first ontology graph labeled with the domaincategory “video games” and a second ontology graph labeled with thedomain category “health activities,” some embodiments may determine thatno associations exist between the two ontology graphs. After obtaining asecond set of documents, some embodiments may determine that a firstn-gram “VR rhythm game” encoded in the first ontology graph isassociated with a second n-gram “cardiovascular exercise,” where thefirst n-gram is encoded in a vertex of the first ontology graph and thesecond n-gram is encoded in a vertex of the second ontology graph. Inresponse, some embodiments may update the first and second ontologygraphs to indicate the detected association, where this detectedassociation may then be used for natural language processing operationsor other operations described in this disclosure. For example, someembodiments may update a set of query generation operations or update aset of categories that are presented in a UI to indicate a detectedassociation between a first n-gram and a second n-gram.

III. Ontology Integration for Summarization

Summarizations of text documents may be used to provide usefulinformation in time-critical scenarios. Additionally, summarizationsprovide the practical benefit of reducing cognitive load on users duringa search operation through natural-language text by providing users withrelevant information that helps them determine which documents toanalyze and which documents to ignore. However, summarization operationsthat do not consider a user's areas of expertise may provide aninappropriate amount of information for a user. For example, a summarythat uses jargon or technical terminology outside of a user's area(s) ofexpertise may be technically relevant but practically inadequate for thegoal of providing a user with the information they need to determine ifa document should be read or otherwise used. Without adequateconsideration for a user's domains, domain classes, or other domaincategory station values, some embodiments may provide document summariesthat are either too simplistic or too technical for a user to interpret.

Some embodiments may use associations between different ontologies asdescribed above to generate text summaries using extractive orabstractive text summarization methods. Some embodiments may use theassociations between different ontologies to generate summarizations oftext or other data stored in corpora of text. After obtaining a queryfrom a user and a set of context parameters associated with the user,some embodiments may retrieve a natural language document using one ormore operations described elsewhere in this disclosure. Some embodimentsmay then generate a text summarization of the retrieved document(s)based on n-grams of the document(s) associated with one or moreontologies. Some embodiments may use indices, ontologies, or another setof associations relating n-grams of one domain category value withn-grams of another domain category value to generate a text summary. Forexample, some embodiments may use an index to directly determine whichn-grams of one domain class found in a query may map to an n-gram ofanother domain class, where the index may be generated from associationsof different ontologies. Alternatively, or in addition, some embodimentsmay traverse edges of different ontology graphs to select n-grams of afirst ontology graph based on n-grams of a second ontology graph.

By generating summaries using ontologies associated with domains orcategories of domains, some embodiments may provide more domain-specifictext summarizations. Incorporating domain-specific ontologies togenerate summarization may result in the generation of more meaningfulor interpretable summarizations that are more likely to retrieveinformation relevant to a query. Additionally, some embodiments may useontology-determined indices to increase the speed or efficiency ofontology-specific summarization generation.

FIG. 13 is a flowchart of an example process by which a domain-specificsummarization may be provided based on a query, in accordance with someembodiments of the present techniques. In some embodiments, the process1300 may include obtaining a query, as indicated by block 1304. Theprocess of obtaining a query may include one or more operationsdescribed above, such as one or more operations described for block 404.For example, some embodiments may obtain a query during a login sessionbetween a client computing device and a server or other computer system,where a set of account parameters of a user account of the session maybe available and included in a set of context parameters. Alternatively,or in addition, the query made by a user may be used to generate one ormore predicted values that may be included in the set of contextparameters.

In some embodiments, the process 1300 may include obtaining a set ofontology graphs, as indicated by block 1310. As described elsewhere, theset of ontology graphs may be stored in various forms and loaded into acomputer system in various forms. For example, some embodiments mayobtain a set of ontology graphs encoded in the form of a set of linkedarrays and lists storing vertices and edges connecting the verticestogether. As described elsewhere in this disclosure, in someembodiments, each respective vertex of a set of vertices of an ontologygraph may be identified by or otherwise mapped to by a respectivelearned representation (e.g., a respective embedding vector in anembedding space) of an n-gram. Alternatively, in some embodiments, eachrespective vertex of a set of vertices of an ontology graph may beidentified by or otherwise mapped to by a respective n-gram, directly.Some embodiments may include a plurality of ontology graphs, where eachontology graph is associated with a different domain of knowledge, adifferent domain class within the domain of knowledge, or other domaincategory values. For example, some embodiments may obtain a firstontology graph associated with a first domain of knowledge and a secondontology graph associated with a second domain of knowledge, where avertex of the first ontology graph identifying an n-gram may map to avertex of the second ontology graph via a cross-graph edge.

Some embodiments may load one or more ontology graphs from a persistentmemory into a non-persistent memory based on a set of user-specificcontext parameters that indicates a domain, class within the domain, oranother domain category value. For example, some embodiments may load aset of values representing a set of graph vertex identifiers, an arrayof edges associating different graph vertices of the ontology, or a setof vectors representing n-grams. Furthermore, some embodiments mayconvert a set of ontology graphs into an index storing pairs of n-gramsthat span between different ontology graphs. In some indices, a pair ofn-grams may span between different ontology graphs if the first n-gramof the pair is part of a first ontology and the second n-gram of thepair is part of a second ontology. As described elsewhere in thisdisclosure, two ontology graphs may be stored as part of a same datastructure or set of data structures, but be made distinct from eachother based on their respective association with different domaincategory values.

Some embodiments may obtain preference weight(s) that are associatedwith an ontology, where a user may have different preference weights fordifferent ontologies or classes within the ontologies. Some embodimentsmay then select one or more ontologies for use to select n-grams orgenerate summaries based on the preference weights. For example, a usermay be indicated as being capable of accessing a first ontology graphassociated with the domain “billing” and a second ontology graphassociated with the domain “clothing.” Some embodiments may select thesecond ontology graph without selecting the first ontology graph whenperforming one or more operations described in this disclosure for theuser based on “clothing” having a greater preference weight.

In some embodiments, the process 1300 may include obtaining a set ofcontext parameters, as indicated by block 1314. As described elsewherein this disclosure, the set of context parameters may include domains ofknowledge, different class categories representing expertise within adomain, user roles, user permissions, predicted domain classes, valuesof environmental variables, or the like. In some embodiments, theprocess of obtaining the set of context parameters may includedetermining one or more values from data associated with a data sessionbetween a server and a client computer device of a user. For example,some embodiments may determine the domain expertise(s) of a user basedon the account by which the user is using to access the data session.Alternatively, or in addition, some embodiments may determine a set ofuser-specific context parameters based on information determined from ananalysis of a user input. For example, some embodiments may determine apredicted domain class representing an expertise score for a user basedon a set of queries made by the user.

In some embodiments, the process 1300 may include obtaining a set ofnatural language documents and corresponding learned representations ofn-grams of the set of natural language documents, as indicated by block1318. Various operations may be performed, such as those described forthe process 400 above. As described elsewhere, set of natural-languagedocuments may be obtained in the form of a corpus of text from varioussources. For example, some embodiments may obtain a set ofnatural-language text documents from a corpora of natural-language textdocuments after receiving a query, where the query may be updated toinclude n-grams of one or more ontologies. Some embodiments may thengenerate learned representations of words, phrases, or other n-grams ofthe documents using one or more operations described in this disclosure.

As described elsewhere in this disclosure, some embodiments maydetermine a set of embedding vectors of a natural language documentusing a transformer model or other neural network model. These embeddingvectors may represent vectors in an embedding space, where pairwisedistances between respective vector pairs indicate semantic similaritiesbetween the n-grams represented the respective pairs. In someembodiments, these embedding vectors may be generated as part of thehidden state outputs of a layer of a set of neural network layers. Asdescribed elsewhere in this disclosure, one or more models may be usedto generate embedding vectors for words or other n-grams, such as BERT,XLNet, GPT, or the like. For example, some embodiments may use XLNet oranother autoregressive transformer model to generate word embeddings,where XLNet is described in Yang et al. (Yang, Z., Dai, Z., Yang, Y.,Carbonell, J., Salakhutdinov, R. R. and Le, Q. V., 2019. Xlnet:Generalized autoregressive pretraining for language understanding. InAdvances in neural information processing systems (pp. 5753-5763)),which is incorporated herein by reference. For example, some embodimentsmay to generate embeddings for a word of a document, where the word (oranother n-gram) may be assigned an embedding vector based on both theword itself, its position with respect to other words surrounding theword, and the embedding vectors of the other words.

Some embodiments may determine different embedding vectors or otherlearned representations for a same n-gram based on other n-grams of thedocument being analyzed or a domain category value associated with auser. For example, some embodiments may generate a first vector for ann-gram of a document when a first user is retrieving the document andgenerate a second embedding vector different from the first embeddingvector for the same n-gram of the document when a second user isretrieving the same document. In some embodiments, the embedding vectorof the word or its surrounding words may be influenced based on anontology graph or plurality of ontology graphs, where the set ofontology graphs may be selected based on a domain category value(s)associated with a user. For example, an association between verticescorresponding with the ontology graphs may be used to update anembedding vector or its distance with another embedding vector. In someembodiments, the update may reduce the distance between relatedembedding vectors. For example, if an ontology graph edge indicates thatan ontology vertex pair mapped to by the embedding vector pair areequivalent to each other or that one is a superset of another, someembodiments may determine or update one or both of the a pair ofembedding vectors to reduce the distance between the vectors.

In some embodiments, the process 1300 may include determining scores fortext sections of a natural-language text document, as indicated by block1320. A text section may include an n-gram or a plurality of in-grams.For example, a text section may include a portion of a word, a wholeword, a phrase, a clause, a sentence, a paragraph, multiple paragraphs,or the like. For example, some embodiments may segment a naturallanguage document into a sequence of text sections including a firsttext section representing a first sentence or paragraph and a secondtext section representing the following sentence or paragraph of thefirst sentence. As described further below, some embodiments may selecttext sections from the sequence of text sections based on their scoresand use the n-grams of the selected text sections to generatesummarizations. Furthermore, as described elsewhere in this disclosure,some embodiments may use a set of learned representations of anatural-language text document determined using another operationdescribed for one or more blocks of the process 1300 to determine scoresfor text sections of the natural-language text document. For example,some embodiments may have determined embedding vectors for words,phrases, or other n-grams of a document during a previous operation anduse the sequences of embedding vectors corresponding to sentences whendetermining scores for the sentences of a document.

Various types of scoring models may be used to score n-grams of anatural-language text document, where n-grams of a natural-language textdocument may be scored by individually scoring the n-grams or by scoringtext sections including the n-grams. In some embodiments, the scoringmodel may be a model used for extractive summarization methods, whereone or more text sections may be selected as summarizing text sectionsbased on the value of the corresponding scores. Some embodiments maydetermine topic scores for each sentence of a natural language document.In some embodiments, each respective topic score corresponding to arespective text section and may indicate relevance to a topic, where thetopic may be determined from a query or an updated query. For example, afirst topic of a query may include the phrase “atrial fibulation” basedon the query including the phrase “atrial fibulation,” and a secondtopic of the query may include the acronym “NVAF” based on a set ofcross-graph associations between the n-gram “NVAF” and the n-gram“Atrial Fibulation.”

In some embodiments, one or more probabilistic models may be used toscore a text section to determine relevance with a document or a queryused to retrieve the document. Some embodiments may use a latentDirichlet allocation (LDA), latent semantic analysis (LSA), or the like.For example, some embodiments may generate topics for a document basedon a LDA model or other probabilistic model and then determine the topicscores of text sections of the document based on the selected topics.Various operations may be performed when determining topics using an LDAmodel, such as representing a document as a set of features using abag-of-words or determining a distribution parameter of a set ofdocuments. For example, some embodiments may determine a distributionparameter based on the frequency of a set of words appearing in a set ofdocuments. Some embodiments may then determine the probability of a textsection being relevant to a specified topic based on a frequency ofmentioning the topic, where the topic may be mapped to by a queryprovided by the user via an ontology graph.

In some embodiments, the scoring model may include a neural network fordetermining a topic score or other type of score indicating that asection of text is relevant for summarization. For example, someembodiments may use a recurrent neural network to determine a learnedrepresentation of a sentence with respect to a specific topic, wheredifferent RNNs may be used to determine different sentence scores for asame sentence with respect to different topics. Some embodiments mayselect a set of text sections that satisfy a score criteria such as arelevance threshold or select a set of text sections based on theirrankings to determine which text sections to analyze or display in auser interface (UI).

In some embodiments, as described elsewhere in this disclosure, someembodiments may generate scores for individual n-grams of a document.For example, some embodiments may generate a score for each n-gram of asentence of a document, and text sections comprising the sentence may bescored based on the individual scores of the n-grams. In someembodiments, the scoring model may include a neural network thatdetermines a sentence score. For example, some embodiments may use arecurrent neural network to determine a learned representation of asentence with respect to a specific topic, where different RNNs may beused to determine different sentence scores for a same sentence withrespect to different topics. Some embodiments may then select a set oftext sections that satisfy a score criteria such as a relevancethreshold (e.g., in the form of a minimum score threshold) or select aset of text sections based on their rankings to determine which textsections to analyze or display in a user interface (UI). In someembodiments, the neural network may have been trained to update a scorein response to detecting the presence of one or more n-grams associatedwith an ontology graph.

In some embodiments, the process 1300 may include selecting an initialset of n-grams based on the natural-language text document and a firstontology graphs of the set of ontology graphs, as indicated by block1324. As described elsewhere in this disclosure, some embodiments mayselect the initial set of n-grams based on the selected text sectionsdescribed above for block 1320 such that each n-gram of the initial setof n-grams is found in a selected text section. Alternatively, someembodiments may select an initial set of n-grams from the entirety of anatural-language text document instead of retrieving n-grams only fromselected text sections. Some embodiments may select the initial set ofn-grams based on each n-gram of the initial set being mapped to one of afirst set of vertices of a first ontology graph, where each of the firstset of vertices map to a concept or other vertex of another ontologygraph. may determine a set of embedding vectors in an embedding spacefor n-grams and determine vertices of an ontology graph identified bythe embedding vectors. For example, some embodiments may determine thata set of five words are in a first ontology and mapped to concepts of asecond ontology and, in response, add the set of five words to theinitial set of n-grams.

In some embodiments, the process 1300 may include determining whether aset of domain-specific indices is available, as indicated by block 1330.Some embodiments may retrieve a set of associations to perform one ormore operations described in this disclosure, where the set ofassociation may include a set of domain-specific indices or a set ofontology graphs. In some embodiments, the set of domain-specific indicesmay include an index having a data structure optimized for informationretrieval, such as a self-balancing search tree or a trie, where theindex may be generated or updated based on a set of ontology graphs. Asdescribed elsewhere in this disclosure, some embodiments may use anindex to determine associations between different n-grams, where thedifferent n-grams may be associated with different domains, differentdomain classes, or other different domain category values. Someembodiments may determine that a cross-domain index is available afterfinding an index storing n-grams or their corresponding embeddingvectors of a first ontology, where the index includes, for each n-gramor corresponding embedding vector(s), an association with other n-gramsor corresponding embedding vectors of a second ontology. If adetermination is made that a cross-domain index that includes theinitial set of n-grams is available, operations of the process 1300 mayproceed to block 1334. Otherwise, operations of the process 1300 mayproceed to block 1344.

In some embodiments, the process 1300 may include selecting one or moreindices based on the set of domain category values, as indicated byblock 1334. Some embodiments may use a single index that includes one ormore keys based on an n-gram, an embedding vector determined from ann-gram, or a domain category value and has, as a value a correspondingn-gram of a different domain category value. Information associated withthe index may indicate that the index provides a mapping from a firstn-gram or its corresponding learned representation to a second n-gram orits corresponding learned representation. Some embodiments may use thisinformation after determining that a user is associated with a domain,domain class, or other domain category value that is mapped to by theindex.

In some embodiments, the process 1300 may include retrieving a secondset of n-grams related to the initial set of n-grams based on the one ormore selected indices, as indicated by block 1338. As describedelsewhere in this disclosure, an index may be stored in various formsthat increase the speed of data retrieval, such as in the form of aself-balanced search tree, a trie, or the like. In some embodiments, aself-balanced search tree, prefix tree, or other index may be loadedinto a cache memory to increase data retrieval speeds, where a cachememory may include an L1 cache, L2 cache, L3 cache, or another cachememory of a different or mixed cache level. A cache memory may refer toa hardware cache that is integrated with a computer processor andcharacterized by being faster to access than other memory of a computersystem and may include one or more SRAM components. By allocatingontology-specific indices into a cache memory of a computing device,some embodiments may accelerate the speed by which ontology-specifictext summarization is performed.

Various operations may be performed to retrieve related n-grams of aninitial set of n-grams using an index. Some embodiments may searchthrough a self-balancing search tree based on a key, where the key maybe an n-gram or a learned representation of the n-gram. Some embodimentsmay search through the self-balancing search tree by starting at a rootof the self-balancing search tree and recursively traversing tree nodesusing the key to retrieve a second n-gram or corresponding embeddingvector at a leaf node of the self-balancing search tree. Alternatively,or in addition, some embodiments may use an index stored in the form ofa trie, where the trie may be associated with a first ontology and asecond ontology such that it may be retrieved from a database or otherdata structure with identifiers of the first and second ontology. Someembodiments may traverse nodes of the trie based on an n-gram of theinitial set of n-grams to retrieve a second n-gram, where the secondn-gram may be part of a different ontology. By using an index connectingn-grams or representations of n-grams between different ontologies, someembodiments may accelerate the speed of data retrieval, textsummarization, or other operations described in this disclosure.

In some embodiments, the process 1300 may include determining a firstset of vertices of a first ontology graph based on the initial set ofn-grams, as indicated by block 1344. As discussed elsewhere in thisdisclosure, some embodiments may include a one-to-one mapping of n-gramsto vertices of an ontology graph. Alternatively, or in addition, someembodiments may map an n-gram to multiple vertices of an ontology graphbased on embedding vectors or other learned representations of ann-gram. Some embodiments may map the first n-gram to multiple verticesby mapping the n-gram to multiple embedding vectors, where eachembedding vector may correspond with a vertex of an ontology graph. Asdiscussed above, some embodiments may convert the initial set of n-gramsinto a set of embedding vectors when selecting the initial set ofn-grams, scoring text sections. For example, some embodiments may havepreviously determined the embedding vectors corresponding to a set ofn-grams and re-use embedding vectors of the n-grams to select verticesof a first ontology graph. Alternatively, or in addition, someembodiments may generate a new set of embedding vectors that areindependent of other embedding vectors that may have been previouslydetermined.

In some embodiments, the process 1300 may include determining a secondset of n-grams and a corresponding second set of vertices of a secondontology graph based on a direct association or indirect associationwith the first set of vertices, as indicated by block 1348. As describedelsewhere in this disclosure, some embodiments may include a pluralityof ontology graphs associated with different domains of knowledge ordifferent class values within a domain of knowledge. For example, someembodiments may retrieve a plurality of ontologies that include a firstontology graph may be associated with the domain “cardio neurology” andthe domain class “3,” which is selected from a list of domain classes[“1”, “2”, “3”]. The plurality of ontologies may also include a secondontology graph that is associated with the same domain of “cardioneurology” but differ by labeled with a domain class “2.”

As discussed elsewhere in this disclosure, some embodiments mayassociate vertices of a first graph with vertices of a second graph viaa direct association. In some embodiments, a direct association betweentwo ontology vertices may include an ontology graph edge represented bya pair of values linking the two vertices by their corresponding uniqueidentifiers. In some embodiments, one or more vertices of the secondgraph may represent a set of concepts that represent supersets, subsets,equivalencies, or other relationship types with an n-gram indicated bythe vertices of the first graph. For example, some embodiments mayrecognize the n-gram “nursing mother” as a named entity associated witha first vertex of a first ontology by directly identifying the vertex bythe named entity or by an embedding vector representing the namedentity. Some embodiments may then detect an association between “nursingmother” and the second n-gram sequence “nursing woman” via an edgeconnecting the first vertex and a second vertex of a second ontologygraph. After detecting the association, some embodiments may include thesecond n-gram in a set of related n-grams of the n-gram “nursingmother.”

In some embodiments, the edge may be labeled with a relationship type,such as a category selected from a list of categories. For example, someembodiments may determine that a first vertex of a first ontology mappedto from the term “nursing mother” in a query is connected to a secondvertex of a second ontology via an edge represented by an ordered pairof vertex identifiers. The edge may be labeled with a categoryindicating a relationship type selected from the list of categories‘[“subset”, “superset”, “cause”, “symptom”].’ For example, someembodiments may determine that the term “nursing mother” is a type of“nursing woman” based on a relationship type associated with a graphedge connecting the vertices corresponding with “nursing mother” and“nursing woman,” where the relationship type may be “subset.” Someembodiments may limit associations between related vertices to specificrelationship types. For example, some embodiments may determine that arelationship type between a first vertex and a second vertex iscategorized with the value “subset” and determine that the relationshiptype is in a set of criteria relationship types. In response to therelationship type being in the set of criteria relationship types, someembodiments may then add the vertex to the second set of vertices foruse when generating a sequence of n-grams, as further described below.Otherwise, some embodiments may ignore the edge associating the firstvertex with the second vertex.

In some embodiments, the process 1300 may include generating a sequenceof n-grams based on the initial set of n-grams, the second set ofn-grams, or the selected text sections, as indicated by block 1360.Generating the sequence of n-grams may include using one or moreabstractive text summarization models, where an abstractive textsummarization model may take input text to generate text summaries thatinclude words or word combinations not found in the input text. In someembodiments, the abstractive text summarization model may include asequence-to-sequence RNN model, where a sequence-to-sequence RNN modelmay use an encoder neural network of the RNN to provide a set of hiddenstate values from a sequence of n-grams. The hidden state values mayinclude a set of learned representations, such as those described above.One or more layers of a first set of neural network layers of an encoderneural network may obtain, as an intermediate input, a set of hiddenstate values outputted from a previous layer of the first set of neuralnetwork layers, where a first input to the RNN model may include asequence of n-grams or learned representations of the n-grams. Forexample, some embodiments may provide a sequence of n-grams to an RNN inorder to obtain a set of hidden values as outputs. Some embodiments mayuse the set of hidden values as inputs for a set of decoder neuralnetworks of the sequence-to-sequence RNN to determine a sequence ofoutput embedding vectors that may then be converted into a sequence ofn-grams for use as a summary.

Some embodiments may use the embedding vectors or other learnedrepresentations determined in an operation described above as part ofthe set of hidden values or otherwise include the embedding vectors inthe set of hidden values. For example, some embodiments may havedetermined a set of embedding vectors based on the n-grams when firstretrieving a document and generating scores for text sections of thedocument. This set of embedding vectors may then be re-used whenselecting vertices of ontologies or generating a sequence of n-grams.Alternatively, or in addition, some embodiments may generate a set ofhidden values that are independent of a previously-calculated the set ofembedding vectors when generating a text summarization.

Some embodiments may use a neural network model to determine words of asummarization string based on a set of hidden state values of the neuralnetwork model. For example, some embodiments may use a decoder neuralnetwork to determine a word of a summarization string based on theoutput of an encoder neural network, where the word being determined maybe the last word of an n-gram sequence. Some embodiments may augmentsummarization operations by using a set of attention values or othervalues determined in combination with the set of hidden values, wherethe set of attention values may indicate which n-grams of a sequence ofn-grams should be given a greater weight when determining an outputusing an RNN.

As described elsewhere in this disclosure, some embodiments may use atransformer model to generate a summarization by determining a set ofpositional encoding vectors, where each respective positional encodingvector may be based on a position of a respective word in the textsection. An encoder neural network of the transformer model may includea multi-headed attention model for performing self-attention operations.As further described below, performing self-attention operations mayinclude assigning attention values to each of the n-grams in a sequenceof n-grams based on the positional encoding vectors and otherintermediate outputs of the encoder neural network model. By performingself-attention operations, some embodiments may assign attention valuesto each word based on relations between different n-grams in thesequence of n-grams. These attention values may then be used asadditional inputs in conjunction with a sequence of embedding vectors bya set of decoder neural network layers to predict additional n-grams fora sequence of n-grams, where the additional n-grams may be at the end ofthe sequence and usable in a summary. Alternatively, or in addition,these attention values may be used to determine the set of embeddingvectors.

Some embodiments may perform self-attention operations by computing aset of key vectors, query vectors, or value vectors determined using aset of embedding vectors and positional encoding vectors. In someembodiments, the key, query, and value vectors may be determined duringa training operation of a transformer model. After training, someembodiments may compute attention values using a function that takes, asinput(s), the sets of key, query, and value vectors. For example, usingan attention-determining function may include computing a product of afirst element of a query vector with a second element of a key vector,where the product may be computed as part of a dot product determinationbetween a query vector of a first n-gram with key vectors of othern-grams of a sequence of n-grams, where the output of the dot productmay be further processed to determine an attention value. Variousmodifications to the output vector(s) may be performed, such asdetermining a root of the output, performing a normalization of the rootby performing a set of softmax operations, or the like. Performing theset of softmax operations may include determining a ratio of anexponential value and a sum of exponential values, where the inputs ofthe exponential value may include outputs of a previous set of neuralnetwork layers, such as the set of decoder neural network layers.Additionally, some embodiments may determine an attention value based onan association an ontology graph. For example, some embodiments mayincrease an attention value based on the attention value being assignedto an n-gram of a second ontology, where the n-gram of the secondontology represents a concept that is mapped to by an initial n-gram ofa query.

Some embodiments may perform abstractive summarization to generate asequence of n-grams by using a pointer generation network model. Using apointer generation network model may include computing a first valuewhen determining an n-gram based on a vocabulary score distribution anda second value based on an attention value distribution. In someembodiments, scores of the vocabulary score distribution may be linkedto a set of n-grams, where the score may be increased for n-grams foundin a user-provided query or associated with an ontology graph vertex.Using the pointer generation network model may include determining an-gram selection score based on a weighted sum that includes the firstvalue and the second value, where the n-gram selection score may then beused to select an n-gram. Using the pointer generation network model mayinclude determining a n-gram selection score associated with an n-grambased on a weighted sum that includes the first value and the secondvalue. Some embodiments may then determine an n-gram based on the n-gramselection score and add the n-gram to a summary. For example, an n-gramselection score for a first n-gram may be defined as a sum of theprobability that a new n-gram is generated after it is weighted by(e.g., multiplied by) by the vocabulary score of the first n-gram andthe sum over a set of attention values corresponding with the word,where the sum over the attention value may be weighted by a complementof the probability that a new n-gram. Some embodiments may determine aprobability value using a random or pseudorandom operation, wheresatisfying an n-gram generation threshold determined from the weightedsum with the probability value may cause the generation of a new n-gramin place of an existing n-gram of a document when determining whatn-gram to use while generating a sequence of n-grams. In someembodiments, the new n-gram may be an n-gram associated with a secondontology graph, where the second ontology graph may be associated with auser domain or domain class.

Some embodiments may also compute a coverage loss value when generatinga sequence of n-grams for a text summary. Some embodiments may perform acoverage loss computation by determining a coverage vector based on aposition and a sum of previous attention vectors. Each set of the setsof previous attention vectors may be associated with a position in thesummarization. For example, some embodiments may determine an attentiondistribution having attention values for each n-gram of a summarization,where the attention values may be determined using one or moreoperations described above. Some embodiments may perform one moreoperations described above to determine a plurality of attentiondistributions, where elements of a respective attention distribution ofthe plurality of attention distributions may be used to update hiddenstate outputs of one or more neural network layer corresponding withn-grams of a sequence of the natural-language text document. Someembodiments may then determine a coverage vector for the next word bycomputing a sum of the attention distributions for each previous n-gramof a summarization. The coverage loss value of an n-gram may be set tothe lesser between the attention value associated with the n-gram andthe coverage value in the coverage vector of the n-gram. Someembodiments may use this coverage loss value as part of a loss functionto determine the next word for a summarization. By incorporating acoverage loss value into the loss function, some embodiments maydecrease the repetitiveness of a summary or otherwise increase theconciseness of the summary.

Some embodiments may be capable of retrieving one or more storedconfigurations or versions of a text summarization model, where thestored configurations may include different sets of neural networkparameters. For example, some embodiments may retrieve two or more textsummarization models for generating a sequence of n-grams based on auser being associated with two different domains or domain class values.Some embodiments may then select which of the text summarization modelsto use based on a preference weight, where the preference weight may bea binary value, categorical value, or categorical value. For example,some embodiments may retrieve a first set of neural network parametersfor a text summarization model in response to a determination that afirst user is associated with a first domain category value.Additionally, some embodiments may then retrieve a second set of neuralnetwork parameters for the text summarization model in response to adetermination that a second user is associated with a second domaincategory value.

Some embodiments may generate an initial sequence of n-grams based on aninput natural-language text document or a corresponding set of embeddingvectors. Some embodiments may then update the initial sequence ofn-grams using a set of related n-grams determined via a set of graphedges of vertices associated with different domain category values. Forexample, some embodiments may select a subset of n-grams of the initialsequence of n-grams. Each respective first n-gram of the subset ofn-grams may be mapped to a respective first vertex of a first ontologygraph that is itself associated with a respective second vertex of asecond ontology graph via a set of associations (e.g., an index, apointer stored as a part of a vertex, or the like). Some embodiments maythen directly replace the respective first n-gram with a respectivesecond n-gram identified by the respective second vertex or otherwiseupdate the initial sequence of n-grams with the respective secondvertices. In some embodiments, replacing the respective first n-gramwith the respective second n-gram may include replacing a respectivefirst embedding vector corresponding with the respective first n-gramwith a respective second embedding vector corresponding with therespective second n-gram. As described further below, some embodimentsmay then present the summarization with the respective second n-graminstead of the respective first n-gram.

In some embodiments, the process 1300 may include presenting a summaryin a UI that includes the sequence of n-grams, as indicated by block1370. Presenting the UI may include one or more operations describedelsewhere in this disclosure, such as operations disclosed for block430. In some embodiments, the UI may include a set of UI elements that,when interacted with by a user, may indicate a feedback message providedby the user. The feedback message may be used to adjust a preferenceweight associated with an ontology graph. By adjusting the preferenceweights, some embodiments may modify the degree to which a specificontology is used when generating a summary. For example some embodimentsmay receive a feedback message indicating that a summary is accurateand, in response, some embodiments may increase a preference weightassociated with the set of ontologies used to generate the summary.Alternatively, some embodiments may receive a feedback messageindicating that a summary is inaccurate and, in response, someembodiments may decrease a preference weight associated with the set ofontologies used to generate the summary.

As described above, some embodiments may adjust preference weightsassociated with different ontology graphs, where the preference weightsmay be used to determine the vocabulary used to generate summaries. Forexample, after adjusting a set of weights associated with a first andsecond ontology, some embodiments may select the second ontology amongstthe plurality of ontologies based on the second weight being greaterthan the first weight. By updating the ontologies used, some embodimentsmay provide a more comprehensible summarization for a user. For example,if a user is associated with a first ontology that is labeled with thedomain category value “expert” and provides a feedback messageindicating that this domain category value is too difficult, someembodiments may reduce the preference weight associated with the firstontology and a subsequent summarization operation may rely on anontology that is labeled with the domain category value “beginner.”

In some embodiments, the UI may visually indicate words, phrases, orother n-grams of a summarization. In some embodiments, the UI mayindicate words of an extracted summarization that match or are otherwisesimilar to words used in the query. Alternatively, or in addition, theUI being presented by a client computer device may indicate that a word,phrase, or other n-gram of a summarization is an n-gram of a vertexassociated with an ontology associated with a user's domain, domainclass, or other domain category value. For example, some embodiments maydisplay a summary on a visual display of a client computer deviceincluding the phrase, “the AF was successful,” where the acronym “AF”may be part of an ontology associated with a domain class of the userthat was not originally in the document being summarized.

Some embodiments may visually indicate the acronym “AF” or other relatedn-grams of a set of related n-grams of a first n-gram retrieved from adocument. Visually indicating a related n-gram may include using one ormore types of visual indicators, such as changing the text color, textsize, text background color, font style, or the like of the acronymrelative to other n-grams of the summarization. Some embodiments mayalso update the summary to include links to other n-grams or otherinformation. Some embodiments may present the UI as a web document,where the source code of the web document may include embedded tagssurrounding a first n-gram, where the presence of the embedded tags maymake the text representing first n-gram an interactive UI element andcause the display of another n-gram mapped to the first n-gram after aninteraction with the interactive UI element. For example, someembodiments may add embedded tags in the vicinity of the acronym “AF”that causes the UI to display the n-gram “fibrillation operation” andfurther cause the UI to display a definition for the concept representedby the n-gram “fibrillation operation” in response to a user clicking onor tapping on the acronym “AF.” Some embodiments may generate the linkbased on the association between the “AF” and “fibrillation operation”via a mapping between a vertex of a first ontology graph identifying then-gram “AF” and a vertex of a second ontology graph identifying then-gram “fibrillation operation.” Alternatively, or in addition, someembodiments may provide a set of UI elements to update associationsbetween n-grams of a set of ontology graphs. For example, someembodiments may permit a user to highlight a set of n-grams and indicatethat the highlighted set of n-grams is associated with another n-gram(e.g., an n-gram of a query, an n-gram of a document, an n-gram that isentered into a text box, or the like).

As described above, some embodiments may present multiple summaries of asame document. For example, some embodiments may generate a first textsummary of a document using a plurality of ontologies and a second textsummary of a document using only one ontology or no ontologies. Thefirst text summary may include n-grams of a first ontology and n-gramsof a second ontology, while the second text summary may include n-gramsof the first ontology without including n-grams of the second ontology.Some embodiments may concurrently display both the first and second textsummaries, where a user may select which type of text summary they wouldprefer to view in a UI. Some embodiments may include the option for auser to concurrently see both text summaries of a document or view onlyone text summary of the document.

FIG. 14 is a flowchart of an example process by which a domain-basedsummarization model may be configured, in accordance with someembodiments of the present techniques. Operations of the process 1400may begin at block 1410. In some embodiments, the process 1400 mayinclude obtaining a corpus of natural text documents, as indicated byblock 1410. Operations to obtain a corpus of natural documents mayinclude one or more operations described in this disclosure. Forexample, some embodiments may obtain a set of text documents from publicor nonpublic sources, where the text documents may be stored inassociation with specific domains, domain classes, or other domaincategory values.

In some embodiments, the process 1400 may include selecting a set oftraining documents and set of training summaries, as indicated by block1420. Some embodiments may select a set of text documents for trainingpurposes, where different subsets of text documents may be associatedwith each other. For example, some embodiments may obtain a first textdocument representing the body of a research article, a second textdocument representing the abstract for the research article, and a thirdtext document representing a protocol derived from the research article.Some embodiments may then use the first and third text document inconjunction to train a text summarization model to generate a summarydetermined from the abstract.

As described above, some embodiments may train and use a plurality ofsummarization models. In some embodiments, each summarization model ofthe plurality of summarization models may be labeled with or otherwiseassociated with different domains of knowledge. For example, someembodiments may train a respective summarization model by using arespective set of training documents labeled with a respective domain ofknowledge as training inputs. After obtaining a query and identifyingthe respective domain based on a user context parameter, someembodiments may then retrieve the respective summarization model andcorresponding model parameters (e.g., neural network parameters,statistical model parameters, or the like) associated with therespective domain.

Some embodiments may obtain a set of text documents and perform one ormore operations to extract a respective pre-existing text summary fromeach respective document of the set of text documents. For example, someembodiments may retrieve a training document from a set of trainingdocuments and segment the training document based on headers orwhitespace separation to obtain an abstract of the document. Afterextracting the abstract, some embodiments may use the abstract as asummary for the text document and use sequences of n-grams of theabstract as a training summary usable to train one or more textsummarization models when paired with a corresponding training document,as described further below. For example, some embodiments may extract aparagraph from a text document in response to a determination that theparagraph has the header “article summary” or is separated by linebreaks from other text in the document. Some embodiments may then addthe paragraph or sequences of n-grams of the paragraph to a set oftraining summaries usable as training objectives when training a textsummarization model. Some embodiments may then, for each respectivepre-existing text summary, perform one or more operations describedabove to add a respective sequence of n-grams of the respectivepre-existing text summary to a set of training summaries or a learningoperation, as described further below.

In some embodiments, the process 1400 may include performing a set oflearning operations to configure a text summarization model, asindicated by block 1430. Performing a set of training operations mayinclude performing a set of supervised learning operations,semi-supervised learning operations, reinforcement learning operations,or the like. For example, as described elsewhere in this disclosure,some embodiments may generate a learned representation such as embeddingvectors for n-grams of a document.

Some embodiments may train or otherwise configure a plurality of textsummarization models based on different domains or domain classes, orother domain category values. For example, some embodiments may train afirst text summarization model for a first domain “neurology” with adomain class of “expert” and train a second text summarization model forthe same domain “neurology” with a domain class of “intermediate,” wherethe first text summarization model and second text summarization modelmay be different. For example, the first and second text summarizationmodel may differ with respect to a number of neural network layers, theweights of the neurons of the neural network, biases of the neuralnetwork, activation function parameters of the neural network, orarchitecture of the neural networks, or the like. Some embodiments mayuse a provided set of training summaries corresponding to differentdomain classes or other domain category values, where a document may beassociated with a plurality of summaries, each summary being associatedwith a different domain category value. Alternatively, or in addition,some embodiments may train a text summarization model and update theoutput of the text summarization model with ontologies indicated by auser profile after the text summarization model as provided the sequenceof n-grams.

Some embodiments may then select one of the neural network models to usebased on the domain category value associated with a user that is to bepresented with a summary. For example, a first user may have the domaincategory class “expert,” and some embodiments may provide a first textsummary generated by a first version of a neural network model. Someembodiments may then provide a second text summary generated by secondversion of the neural network model for a second user after adetermination that the second user is associated with the domaincategory class “neophyte.”

FIG. 15 is an example user interface including an ontology-generatedsummary, in accordance with some embodiments of the present techniques.The UI 1500 shows a search bar 1510 displaying the query, “nursingmothers and benzoyl peroxide.” After an interaction with the UI element1512, some embodiments may display a first search result box 1520 havinga document summary 1522 and a second search result box 1530 having adocument summary 1532.

Some embodiments may perform one or more operations described in thisdisclosure to generate the document summary 1522 based on the documenttitled “Carcinogenesis, Mutagenesis, Fertility” identified by the firstsearch result box 1520. For example, some embodiments may use anabstractive summarization method to generate the summary, “nocarcinogenicity, photocarcinogenicity, or fertility studies conductedwith EPIDUO FORTE gel.” As described elsewhere in this disclosure, someembodiments may use a set of ontologies to recognize the named entity“benzoyl peroxide” in a first ontology as being associated with thenamed entity “EPIDUO FORTE” of a second ontology graph, where the secondontology graph may be labeled with a domain class value indicated by theuser.

Similarly, embodiments may perform one or more operations described inthis disclosure to generate the document summary 1532 based on adocument titled “8.3 Nursing Mothers,” identified by the second searchresult box 1530, which is shown to be displayed concurrently with thedocument summary 1522. Some embodiments may generate the documentsummary 1532 using the same text summarization model as the one used forgenerating the document summary 1522. For example, some embodiments mayuse a text summarization model to search through a set of ontologies orindices representing the ontologies to determine a first set of n-gramsof one or more domains or classes of domains indicated by a userprofile. Some embodiments may use n-grams or learned representations ofthe n-grams indicated by the set of ontologies or indices as mapped toone or more n-grams of the query to generate the document summary 1522and the document summary 1532.

Some embodiments may update the UI 1500 to indicate one or more n-gramsare related to an n-gram of a query via a set of ontology graph edges.For example, the named entity “EPIDUO FORTE” is shown to be bolded inresponse to a determination that “EPIDUO FORTE” maps to the term“Benzoyl Peroxide” in an index or cross-graph edge of a set ofontologies. In addition, the n-gram “nursing mothers” is associated viaa cross-graph edge with the n-gram “nursing woman,” where the box 1534may indicate a highlighting or color change. In some embodiments, the UI1500 may be presented as a web document, and the embedded tags aroundone or more n-grams determined as related via a set of ontologies may beused convert the text display of the n-gram into a hyperlink orotherwise generate an interactive UI element that overlaps with the textdisplay of the n-gram. In some embodiments, interacting with theinteractive UI element may cause the display of the n-gram that causedthe display of a second n-gram, where the second n-gram may be an n-gramof the query or an n-gram of a concept to which the n-gram of the querymaps.

IV. Question Generation

As discussed elsewhere in this disclosure, some embodiments may use oneor more indices to obtain or process information based on a query. Inmany cases, the query posed by a user may be provided in a formdifferent from that used by a document storing an answer to the query.For example, a query of a user may be written in the natural languageform, “Does it take long to grow E. coli.” Some embodiments may use thequery to search through a corpus of documents to retrieve an answer tothe query in a protocol, where the protocol may recite “Escherichia colirequires 24 hours to incubate.” However, the different words, phrases,and structure of a user-provided query may also provide text from otherdocuments that are not as relevant, reducing both the accuracy andeffective speed of a search operation.

Some embodiments may use an index that matches n-grams of a query withn-grams stored in or otherwise associated with documents (e.g., asmetadata tags). As described above, the index may be constructed using aset of operations that includes scanning the words, phrases, or othern-grams of the text of a corpus of documents and generating a list ofn-grams mapping to the respective document(s) in which they are found.However, using indices that are unable to account for vocabularydifferences or phrasing differences between a user's query and theanswer stored in a document may provide suboptimal search results to thequery. Such differences may include an omission of a word that is partof the answer to the query or the inclusion of extraneous words that arenot present in the answer. Additionally, a query may be syntacticallydifferent from the structure of a document that contains an answerrelevant to the query. Furthermore, some indices may fail to distinguishbetween multiple passages within a document containing the same n-gram,but where the context in which the n-gram is used may be sufficientlydifferent as to cause the n-gram to be mapped to a different embeddingvector.

Some embodiments may accelerate the speed of data retrieval bygenerating queries based on the text in a document, where an identifierof the document or other data associated with the document may be storedin an index mapping the computer-generated query to the document. Forexample, some embodiments may obtain a document from a corpus of text,select a text section based on the likelihood that the text sectionincludes an answer relevant to a user-provided query, and generating aquery based on the text section. The computer-generated query mayinclude words or phrases from the document, where n-grams of thecomputer-generated query or the text section(s) used to generate thequery may be updated or replaced based on a set of ontologies asdescribed elsewhere in this disclosure. After augmenting an index withthe data associated with the computer-generated query, some embodimentsmay then use the index when performing a search for documents based on auser-provided query.

By performing one or more of the operations described in this disclosureto update the index based on the computer-generated queries, someembodiments may provide faster or more accurate search results forqueries. Some embodiments may also increase the accuracy of such searchresults by updating computer-generated queries with alternativeterminology or shared concepts based on a set of ontologies associatedwith different domains. Some embodiments may further re-arrange or formalternative structural constructions of a computer-generated query basedon a set of query structures, such as a query structure formed from ahistory of previous queries. By generating or otherwise updating queriesand storing learned representations of them in an index in associationwith a document or text in the document, some embodiments may accountfor variations in query vocabulary or syntax that may occur innatural-language queries. Such operations may be integrated into achatbot application, voice command application, or the like.

FIG. 16 is a flowchart of an example process by which a query-augmentedindex is generated and used, in accordance with some embodiments of thepresent techniques. Operations of the process 1600 may begin at block1604. In some embodiments, the process 1600 may include obtaining acorpus of natural-language text documents, as indicated by block 1604.Operations to obtain a corpus of natural-language text documents mayinclude one or more operations described above. For example, someembodiments may obtain one or more documents of corpora from a set ofpublic or private data sources, where the text documents may be storedin various formats and with various metadata.

In some embodiments, the process 1600 may include obtaining one or moren-gram sets of a document of the corpus of natural-language textdocuments, as indicated by block 1608. In some embodiments, an n-gramset may include a sequence of n-grams such as a phrase, a clause, asentence, a paragraph, or the like. Alternatively, or in addition, ann-gram set may include a single n-gram or a non-sequential plurality ofn-grams. Some embodiments may perform preprocessing operations onn-grams to increase the accuracy or efficiency of data processing, wheresuch preprocessing operations may include stemming, lemmatizing, orrooting.

Some embodiments may obtain the sets of n-grams by segmenting thenatural-language text documents based on punctuation. For example, someembodiments may obtain the sets of n-grams by segmenting the naturallanguage text documents into sentences, where the segmentation may use aperiod as a delimiting element. Alternatively, or in addition, someembodiments may obtain n-grams from graphical or tabular elements of adocument. For example, some embodiments may obtain one or more n-gramsfrom a document table, where the document table may be displayed as atwo-dimensional table with a set of labeled rows or columns. Someembodiments may perform operations to obtain a table title, row title,row identifier, column title, column identifier, or other table elementsas one or more n-gram sets. For example, some embodiments may determinea column title of a document table and one or more associated tablevalues in the corresponding column, where each value may correspond to adifferent row of the table. As further described below, some embodimentsmay then determine a score based on the n-grams of the tabular data andselect n-grams of the tabular data based on the score.

In some embodiments, the process 1600 may include determining a set ofscores for the one or more n-gram sets, as indicated by block 1612. Someembodiments may determine scores for sets of n-grams, where eachrespective score may quantify or otherwise indicate an importance of therespective n-gram set with respect to indexing operations. Someembodiments may quantify an importance of the respective n-gram setbased on its relevance to a specified set of topics or a relevance to aspecified set of queries. Alternatively, or in addition, someembodiments may quantify an importance of the respect n-gram set basedon its connections to other sets of n-grams in a document.

Some embodiments may determine different types of scores when usingscores to select sets of n-grams, as further described below. Forexample, some embodiments may determine a first set of scorescorresponding to an initial plurality of n-grams sequences that is thenusable to determine a subsequent set of n-gram sequences, where theinitial plurality of n-grams sequences may include a plurality ofphrases, sentences, multiple sentences, or the like. By generating aninitial plurality of n-gram sets and filtering them into a lesser numberof n-gram sets, operations may be made more efficient by reducing thenumber of n-gram sets that some embodiments may process using morecomputing-resource-intensive operations.

Some embodiments may perform a first set of operations to generate aninitial set of scores for the initial plurality of n-grams sequences bydetermining a count of the times by which one or more n-grams of adocument occur in the document. For example, some embodiments maysegment a document into an initial plurality of n-gram sequences, suchas an initial plurality of sentences. The respective n-gram sequences ofthe initial plurality of n-gram sequences may be assigned a respectivescore based on an n-gram count indicating the number of times that therespective n-gram is used. For example, some embodiments may determinean n-gram count for each n-gram of some or all of the n-grams in theplurality of n-gram sequences. Some embodiments may then determine arespective score for a respective n-gram as being equal to the n-gramcount, being a multiple of the n-gram count, or correlating with then-gram count. Some embodiments may then determine an n-gram sequencescore associated with an n-gram sequence by combining the sets of n-gramcounts (or scores based on the sets of n-gram counts) of the n-gramsequence. Combining the counts may include adding, multiplying, using anexponential function, some combination thereof, or the like. Forexample, for each respective sequence score of a set of sequence scorescorresponding with an initial plurality of n-gram sequences, someembodiments may determine a sum of the n-gram counts corresponding withthe n-grams of the respective n-gram sequence.

Some embodiments may modify (e.g., increase or decrease) an n-gram scorein response to a determination that a vertex of an ontology graph mapsto the corresponding n-gram. For example, an n-gram score of an n-gramthat may have been equal to “0.3” may be updated to 0.6″ based on adetermination that the n-gram maps to a vertex of an ontology graphassociated with a first domain. Some embodiments may then select asubset of the initial plurality of n-gram sequences based on adetermination if the corresponding plurality of sequence scores satisfya sequence score threshold. For example, some embodiments may select asentence from a plurality of sentences for further n-gram selectionoperations based on the sentence being associated with a sentence scoregreater than a sequence score threshold. As further described below, theselected subset may then be used as the input set of n-gram sequences(or another set of n-gram sets) for further scoring operations or forother query-generating operations.

Some embodiments may perform operations similar to a Textrank operationbased on n-gram connectivity in an n-gram sequence to determine a set ofn-gram sequences, where Texrank is described by Mihalcea et al.(Mihalcea, R. and Tarau, P., 2004, July. Textrank: Bringing order intotext. In Proceedings of the 2004 conference on empirical methods innatural language processing (pp. 404-411)), which is incorporated hereinby reference. Some embodiments may rank n-grams of a document bydetermining a document-specific vocabulary of n-grams and performingoperations to assign n-gram scores to each n-gram of the vocabulary(“vocabulary n-gram”). Some embodiments may determine a set ofvocabulary n-grams based on the n-grams of a document by adding n-gramsnot already in the vocabulary of n-grams to the vocabulary of n-gramssuch that each n-gram of the vocabulary may be found in the document,where some embodiments may first lemmatize, stem, determine roots for,or otherwise process the n-grams of a document.

Some embodiments may segment the first document into a set of sentencesor other plurality of document n-gram sequences and generate arespective set of n-gram pairs for each respective sentence, where eachn-gram pair includes two n-grams of the sentence. Some embodiments mayselect a sequence of n-grams of the sentence as an n-gram window, wherethe window size may be updated by a processing parameter. For example,some embodiments may generate the two n-gram windows [“man”, “owns”,“large”] and [“owns”, “large”, “tree”] from the n-gram sequence “manowns large tree” after a determination that the n-gram window size isequal to ‘3.’ Some embodiments may then generate a respective subset ofa set of n-gram pairs for each window. For example, some embodiments maygenerate the set of n-gram pairs [“man”, “owns”], [“owns”, “large”], and[“large”, “tree”] from the n-gram window [“man”, “owns”, “large”]. Someembodiments may then update a data table or other data structureidentifying the relationships between n-grams of the n-gram vocabularysuch that every instance of an n-gram pair updates (e.g., increases) thecount. For example, after encountering an instance of the n-gram pair[“man”, “owns”], some embodiments may update a data table storing acount of instances that the n-gram pair [“man”, “owns”] is present inthe document. Alternatively, or in addition, some embodiments maygenerate the set of n-gram pairs from a larger sequence of n-gramswithout using a window. For example, some embodiments may generate a setof n-gram pairs for each sentence of a document, where a pair exists torepresent each pairing of one n-gram of the sentence with another n-gramof the sentence.

Some embodiments may determine an n-gram weight based on the sum ofoccurrences of the n-gram in an n-gram pair, where the values used todetermine the sum may be weighted by the number of other pairs. Someembodiments may then associate a respective n-gram weight with therespective n-gram in the vocabulary of n-grams. For example, someembodiments may determine that a first n-gram is associated with asecond and third n-gram based on a set of n-gram pairs. The secondn-gram may be only associated with the first n-gram and the third n-grammay be associated with the first n-gram and also associated with afourth n-gram, fifth n-gram, and sixth n-gram. The n-gram weightassociated with the first n-gram may be determined based on a sum of theconnections to the first n-gram indicated by the pairs normalized by thenumber of other n-gram connections of each of the other n-grams of thepairs. For example, the association with the second n-gram may add “1”to the n-gram weight for the first n-gram based on the second n-gram notbeing in a pair with other n-grams and the second n-gram may add “0.25”to the n-gram weight of the first n-gram based on the third n-gramsplitting its connection contribution amongst four different n-gramsthat include the first n-gram. Some embodiments may then update eachcontribution to the n-gram weight by an initial weight of the second andthird n-grams to determine the n-gram weight of the first n-gram.Additionally, some embodiments may perform one or more of the operationsdescribed above using linear mathematical operations when determining ann-gram score.

After determining scores for n-grams in the vocabulary of n-grams, someembodiments may then determine a plurality of n-gram sequence scores bydetermining a respective n-gram sequence score for each respectiven-gram sequence of the plurality of n-gram sequences. In someembodiments, the subset of n-gram weights may be associated with thesubset of n-grams that form the respective n-gram sequence. For example,some embodiments may determine a sum of the individual n-gram scores ofa sentence and set the n-gram sequence score to be the sum of thescores. Alternatively, or in addition, some embodiments may performother operations to determine an n-gram sequence score, such asdetermining a product, an exponential value, logarithmic function, somecombination thereof, or the like.

Some embodiments may use a neural network model to select a plurality ofn-gram sets, where the n-gram sets may be sequences of n-grams (e.g.,phrases or sentences). The neural network model may be trained todetermine whether or not a set of n-grams is likely to include an answerto a query based on a training set of sequences. For example, someembodiments may use a feed-forward neural network with a backpropagationmechanism to determine a probability score that a sequence of n-grams ofwould include an answer to a user query. As described elsewhere in thisdisclosure, the model parameters of the neural network may betransferred from a previous data source. Alternatively, or in addition,the model parameters of the neural network may be trained based ondomain-specific data or provided based on a set of domain expertanalysis. Furthermore, an indicated domain category value associatedwith an ontology may be an input of the neural network, where differentprobability scores may be provided for the same sequence of n-grams byusing different domain category values.

As stated elsewhere in this disclosure, some embodiments may use a setof ontology graphs or data related to a set of ontology graphs to modifya set of n-gram weights or other values associated with an n-gram. Forexample, some embodiments may determine whether an n-gram maps to avertex of an ontology graph. In response to a determination that then-gram maps to the vertex, some embodiments may update the weightassociated with the n-gram. For example, some embodiments may update then-gram weight by increasing the n-gram weight. Additionally, someembodiments may store the updated n-gram weight based on adomain-specific criterion. For example, some embodiments may store datain a first index specific to a first domain category value. Someembodiments may then determine whether an n-gram of a document maps to avertex of an ontology graph that is categorized with the first domaincategory value. In response to a determination that the n-gram maps tothe vertex associated with the first domain category value, someembodiments may update the weight associated with the n-gram. Otherwise,some embodiments may leave the n-gram weight unmodified, even if then-gram maps to another vertex of a different ontology graph. By updatingweights based on different ontologies, some embodiments may generatedifferent indexed questions for users associated with different domainsor different domain category values.

As disclosed elsewhere in this disclosure, some embodiments may accessdifferent indices or different portions and index based on a usercontext parameter, such as one identifying a domain category value.Alternatively, or in addition, some embodiments may apply differentscoring systems based on a user context. For example, some embodimentsmay use a first scoring model to determine scores for a set of sentencesof a document, where the scores may indicate a predicted relevance to afirst topic. Some embodiments may then use a second scoring model todetermine a different set of scores for the same set of sentences of adocument, where the second set of scores may indicate a predictedrelevance to a second topic.

In some embodiments, the process 1600 may include selecting one or moren-gram sets based on the set of scores, as indicated by block 1616. Someembodiments may select n-gram sets based on a score threshold, where ann-gram set having an n-gram set score greater than the score thresholdis selected for use when generating a query, as further described below.Some embodiments may select n-gram sets based on a ranking result of theset of scores. For example, some embodiments may determine a score foreach sentence or other n-gram sequence in a plurality of n-gramssequences. Some embodiments may then rank scores and select the greatestor least m scores, where m may be any non-negative integer greater thanzero.

In some embodiments, the process 1600 may include generating a set ofqueries based on the selected n-gram set(s), as indicated by block 1620.As discussed elsewhere in this disclosure, some embodiments may generatetext from other text. Some embodiments may use one or more of themethods described in this disclosure to generate a set of queries basedon n-grams of the one or more n-gram sets selected above, such as one ormore of the text summarization models described above. For example, someembodiments may use a neural network model having one or more attentionmechanism implementations to generate text from a sequence of n-grams.

Some embodiments may generate a query after being provided with a set ofn-grams selected from a document by determining a set of embeddingvectors based on the selected n-grams using a transformer neural networkmodel. For example, some embodiments may use a transformer neuralnetwork model that includes one or more attention mechanisms to generatea query based on n-grams from the sentence, “promise-based architectureare the backbone of the modern internet.” Using a transformer neuralnetwork may include determining a set of attention values based on anattention query value and a attention key value, where the attentionquery and attention key values may be based on a matrix or other datastructure constructed based on similarity scores determined betweenpositions of an input sequence of n-grams or their corresponding learnedrepresentations. Some embodiments may determine a set of attentionvalues based on a random feature map and use the set of attention valuesto generate or update a query. For example, some embodiments maydetermine embedding vectors for the n-grams of the query using anencoder neural network and generate a first random feature map based onthe set of embedding vectors using a feature map function. In someembodiments, using the feature map function may include generating afirst set of random or pseudorandom variables and multiplying at leastone variable of the first set of random or pseudorandom variables withthe at least one element of the set of embedding vectors.

As described elsewhere in this disclosure, some embodiments may performsimilar operations to determine a set of positional encoding vectors anduse the positional encoding vectors in combination with the embeddingvectors to determine a set of attention values. Additionally, someembodiments may update a respective element of a set of attentionvectors based on the attention vector element corresponding with arespective n-gram, where the respective n-gram or its learnedrepresentation is found in an ontology. For example, if an attentionvalue for a first n-gram is initial the value “0.05,” some embodimentsmay determine that the first n-gram maps to a vertex of an ontologygraph and, in response, increase the attention value to “0.07.” Someembodiments may then generate a query using the neural network based onthe set of attention values. For example, some embodiments may use aneural network having neural network layers that use one or more of thesets of attention values as inputs to predict n-grams for a masked setof n-grams. Additional n-grams or to determine new n-grams for use assubstitute n-grams for n-grams of a user-provided query.

Some embodiments may implement transfer learning to increase the speedrequired to train a neural network model and the accuracy of the trainedmodel. Furthermore, some embodiments may use a unified text model whenperforming a plurality of the operations described in this disclosure.For example, some embodiments may use a Text-to-Text Transfer Transform(T5) architecture, such as that described in Raffel et al. (Raffel, C.,Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li,W. and Liu, P. J., 2019. Exploring the limits of transfer learning witha unified text-to-text transformer. arXiv preprint arXiv:1910.10683),which is incorporated herein by reference. For example, some embodimentsmay use a set of learned representations that was first generated forone set of operations, such as text summarization operation, to performanother set of operations, such query generation operations. Using asystem based on the T5 architecture or other unified transfer learningmodel may include using a variety of data types from a previous datasetthat may or may not be similar to a text document being processed by atext generation model. For example, some embodiments may train a textgeneration model that was initialized with a pre-trained model, wheresome embodiments may then perform a reduced-scope training operation forspecific text generation tasks such as text summarization or querygeneration.

Some embodiments may generate multiple queries based on a set ofontology graphs and the natural-language text of a document. Forexample, some embodiments may generate a plurality of queries based onthe phrase “parabolic geometry is useful in this scenario.” Someembodiments may first use one or more natural-language processingoperations to associate the n-gram “scenario” with an n-gram used inanother sentence, such as “distance determination.” Some embodiments maygenerate a plurality of computer-generated natural-language queriesusing one or more of the operations described in this disclosure, wherea first natural-language query recites, “what is parabolic geometryuseful for” and a second natural-language query recites, “what is usefulfor determining distances?”

Some embodiments may generate or update one or more queries based on aset of ontologies in combination with a user context, where a usercontext may indicate a domain, class of the domain, another domaincategory value, or other parameters. Using a context to generate orupdate a query may include performing a query expansion operation asdescribed elsewhere in this disclosure to generate or update the query.For example, some embodiments may update a computer-generated query,“what is parabolic geometry useful for?” based on a first ontology graphcategorized with the domain category value “mathematics.” By referencingthe first ontology graph or an index based on the first ontology graph,some embodiments may recognize the term “parabolic geometry” as mappingto a vertex of the first ontology graph that is associated with a secondvertex of the ontology graph. For example, the second vertex mayrepresent the concept “Euclidean geometry,” where the second vertex maydirectly map to the alternative n-gram “Euclidean geometry.” Someembodiments may then update the first computer-generated query torecite, “what is Euclidean geometry useful for?” By performing a queryexpansion operation that includes generating or updating an n-gram of anatural-language query with a set of alternative n-grams associated withthe other vertices of an ontology via a set of shared concepts, someembodiments may increase the likelihood of generating natural-languagequeries that will be similar to queries provided by a user.

Some embodiments may generate different queries based on a user contextparameter by selecting or updating a text generation model based on theuser context parameter. For example, some embodiments may use a firstneural network model corresponding to a first context parameterassociated with a first type of user profile. The first neural networkmodel may include five encoder neural network layers to generateembedding vectors of n-grams and five decoder neural network layers togenerate a set of decoded n-grams based on the embedding vectors, wherethe decoded n-grams may be used for a computer-generated query. Someembodiments may then use a second neural network model having tenencoder neural network layers and ten decoder neural network layers togenerate queries corresponding to a second context parameter associatedwith a second type of user profile.

Some embodiments may generate or otherwise update one or more queriesbased on a set of n-grams of a summary, such as a summary generatedusing one or more of the embodiments described in this disclosure. Forexample, some embodiments may use an abstractive summarization model,such as a pointer generation network model to generate a summary of adocument. Some embodiments may then generate a query from the summary byperforming one or more of the operations described in this disclosure.For example, some embodiments may segment a summary into a sequence ofn-grams. Some embodiments may then assign scores to each sequence ofn-grams, select queries based on a determination of which sequences ofthe set of n-gram sequences satisfy a sequence score threshold, andgenerate a query based on the selected n-gram sequences. Someembodiments may be able to account for further variations in a query bygenerating multiple queries based on abstractive summaries, which mayinclude phrases or summarizing statements that are not present in thedocument being summarized.

Some embodiments may generate or otherwise update one or more queriesbased on a history of previously-used or previously-generated queries.Some embodiments may access a history of queries via a database, wherethe history may include both the queries and the set ontologies used togenerate or update a query. Some embodiments may then determine avocabulary of the n-grams used to generate the queries (“query n-grams”)and sort the query n-grams based on a parts-of-speech library. Forexample, some embodiments may analyze a history of queries to categorizethe n-grams of the queries into nouns, pronouns, verbs, adjectives, orthe like. Some embodiments may then determine a count of the n-grams andgenerate or update a text-generation model based on the count of n-gramsused. For example, some embodiments may generate a history-specificvocabulary including a first query n-gram or text structure categorizedas a “why” query indicating that a query is requesting information onthe cause of a subject matter. Some embodiments may then performoperations to generate a “why” query by selecting the first query n-gramand replacing an n-gram of a query with the first query n-gram.Alternatively, or in addition, some embodiments may combine differentquery n-grams or different query structures when generating a new queryor update an index based on a computer-generated query.

As described elsewhere in this disclosure, some embodiments may retrievemultiple documents based on a query. For example, some embodiments mayperform one or more of the operations above to obtain another pluralityof n-gram sets of a second document, such as another plurality ofsentences of the second document. Some embodiments may perform one ormore operations described above to determine a second set of scorescorresponding with each n-gram set of the second plurality of n-gramsets, where the second set of scores is determined using a same scoringmodel as the model used to generate the first set of scores. Someembodiments may then select a second n-gram set based on the second setof scores using a set of operations similar to or the same as thosedescribed above. Some embodiments may then use a text generation modelto generate a query based on both a first n-gram set of a first documentand the second n-gram set of the second document. The query may then beused to update an index to map the query to at least one of the firstdocument or the second document.

In some embodiments, the process 1600 may include determining a set oflearned representations based on the set of computer-generated queries,as indicated by block 1624. As discussed elsewhere in this disclosure,some embodiments may determine a set of learned representations such assingular values or vectors in a vector space using one or more learningmodels. For example, as discussed elsewhere in this disclosure, someembodiments may determine a set of embedding vectors for each word orother n-gram of a document. Some embodiments may generate a set ofphrase vectors or a set of sentence vectors for phrases or sentences. Insome embodiments, each type of vector may correspond with a differentvector space. For example, embedding vectors for n-grams may correspondwith a first vector space and sentence vectors may correspond with asecond vector space having a different number of dimensions. Someembodiments may generate a learned representation, such as a vector, torepresent a query. As disclosed further below, some embodiments may thendetermine a similarity score between the learned representation of thecomputer-generated query and a user-provided query.

In some embodiments, the process 1600 may include updating an index tomap a first set of learned representations to the first document, asindicated by block 1628. As discussed elsewhere in this disclosure, someembodiments may use an index, where an index may include a set offeatures that are usable to indicate document content in order toincrease the efficiency of a search operation. Some embodiments maystore the entirety of the computer-generated query in the index orotherwise associate the index-stored value with the text of thecomputer-generated query. Alternatively, or in addition, someembodiments may store a learned representation of the computer-generatedquery in the index. In some embodiments, the index may be stored in theform a set of linked pairs or triplets of values, where each pair ortriplet may be a record of the index that maps different values,identifiers, pointers, or other types of information. Some embodimentsmay update an index to include a record that maps a learnedrepresentation of a computer-generated query with the text position oftext used to generate the query, the first document itself, or anothervalue associated with the first document.

One or more of the updated indices may stored in various other formsthat may increase the speed of data retrieval, such as in the form of aself-balanced search tree, another type of m-ary tree, a trie, or thelike. Furthermore, as described elsewhere in this disclosure, differentindices or different sections of an index may be accessed based on acontext parameter associated with one or more users. For example, someembodiments may include user profiles that categorize users as beingassociated with at least one of three different domains of knowledge,labeled with the list of category values ‘[“infrastructure”,“cybersecurity”, “development”].’ As described elsewhere in thisdisclosure, some embodiments may update an index based on the dataassociated with an ontology categorized with the domain (e.g., based onvertices and edges of an ontology graph). For example, some embodimentsmay determine that a computer-generated query is includes a first n-grammapping to a vertex of an ontology graph labeled with the category“infrastructure,” where the first n-gram is associated via a graph edgeto a second vertex mapping to a second n-gram. Some embodiments may thengenerate a second query that includes the second n-gram. Someembodiments may then update a first index associated with the category“infrastructure” without updating a second index associated with thecategory “cybersecurity,” where updating an index may include generatingor modifying tree nodes of a search tree. For example, some embodimentsmay update a trie to include an additional set of tree nodes thatcollectively store an additional key value and a leaf node storing orpointing to an identifier of a document, where the identifier mayinclude a position of the n-gram sequence used to generate the firstcomputer-generated query.

As described elsewhere in this disclosure, various operations may beperformed to retrieve related n-grams of an initial set of n-grams usingan index. Some embodiments may search through a self-balancing searchtree based on a key, where the key may be an n-gram or a learnedrepresentation of the n-gram. Some embodiments may search through theself-balancing search tree by starting at a root of the self-balancingsearch tree and recursively traversing tree nodes using the key toretrieve a second n-gram or corresponding embedding vector at a leafnode of the self-balancing search tree. Alternatively, or in addition,some embodiments may use an index stored in the form of a trie (i.e.prefix tree), where the trie may be associated with a first ontology anda second ontology such that it may be retrieved from a database or otherdata structure with identifiers of the first and second ontology. Someembodiments may traverse nodes of the trie based on an n-gram of theinitial set of n-grams to retrieve a second n-gram, where the secondn-gram may be part of a different ontology. By using an index connectingn-grams or representations of n-grams between different ontologies, someembodiments may accelerate the speed of data retrieval, textsummarization, or other operations described in this disclosure.

Some embodiments may perform a local search through the document basedon the set of computer-generated queries to retrieve one or moresections of text or other data and then map these sections of text orother data to the computer-generated query. For example, someembodiments may generate a first query “how do I manage apple trees”based on a first n-gram set comprising the sequence of n-grams “appletree management.” Some embodiments may then use the first query toretrieve text data from the document and determine that a first textsection by the starting and ending positions [“15031”, “23162”] of thedocument “doc101x.doc” is most likely to include text relevant to thisquery. Some embodiments may then update the nodes of an index to map alearned representation of the first query “how do I manage apple trees”to the document “doc101x.doc” and the starting and ending positions[“15031”, “23162”].

As described elsewhere in this disclosure, an index such as a prefixtree, self-balanced search tree, m-ary search tree, or the like may beloaded into a cache memory to increase data retrieval speeds whenupdating the index or retrieving data from the index. As describedelsewhere in this disclosure, a cache memory may include an L1 cache, L2cache, L3 cache, where different types of cache memory systems mayindicate different levels of available memory or the speed of memoryaccess. As described elsewhere in this disclosure, some embodiments mayload one or more elements of an index into cache memory in response to adetermination that a user having a user context parameter associatedwith the index is using a computing device to perform one or more of theoperations described in this disclosure.

In some embodiments, the process 1600 may include obtaining a userquery, as indicated by block 1632. In some embodiments, the process ofobtaining a user query may include one or more operations describedabove. For example, some embodiments may obtain a query during a datasession between a client computing device and a server or other computersystem executing one or more operations described in this disclosure.During the data session, a set of context parameters may be availablevia the set of account parameters or other data loaded during the datasession. Similarly, as described above, a query made by a user may beused to generate one or more predicted values that may be included inthe set of context parameters. For example, based on a match between aset of terminology used in a query and a set of terminology of a set ofontologies, a user may be assigned with the domain category values“entomologist” and “expert.”

In some embodiments, the process 1600 may include determining a secondset of learned representations based on the user query, as indicated byblock 1636. Some embodiments may determine a set of n-grams using one ormore operations described in this disclosure. For example, someembodiments may use some or all of the words of the query as n-gramseither with or without a filtering operation(s) to modify the words.Some embodiments may use the same model to determine the second set oflearned representations that was used to determine the first set oflearned representations corresponding with the set of computer-generatedqueries. For example, some embodiments may have used a self-attentiveneural network to determine a first sentence embedding based on acomputer-generated query and then use the same self-attentive neuralnetwork to determine a second sentence embedding based on the query.

Some embodiments may determine a set of n-grams using one or moreoperations described in this disclosure. For example, some embodimentsmay use some or all of the words of the query as n-grams either with orwithout a filtering operation(s) to modify the words. Some embodimentsmay use a first model to determine the second set of learnedrepresentations, where the first model may have been used to determinethe first set of learned representations corresponding with the set ofcomputer-generated queries. For example, some embodiments may have useda self-attentive network to determine a first sentence embedding basedon a computer-generated query and then use the same self-attentivenetwork to determine a second sentence embedding based on the query.

In some embodiments, the process 1600 may include determining asimilarity score based on the first and second set of learnedrepresentations, as indicated by block 1640. A similarity score may beused to indicate a semantic similarity between two learnedrepresentations or their corresponding sequences of n-grams. Someembodiments may determine a similarity score based on a differencebetween a pair of learned representations, such as a pair of integers, apair of vectors, a pair of category values, or the like. For example,some embodiments may determine a first sentence vector based on acomputer-generated query and a second sentence vector based on auser-provided query. Some embodiments may then determine a differencebetween the first and second sentence vectors and determine a distance(e.g., a Manhattan distance, Euclidean distance, another type ofMinkowski distance) between the first and second sentence vectors anduse the difference as a similarity score or otherwise base thesimilarity score on the distance. For example, some embodiments maydetermine a Euclidean distance between the first and second sentencevectors by determining a vector difference of a first and secondsentence vector (i.e., subtracting the first vector from the secondvector) and determining a root of squared sum of the elements of theresulting vector difference. Alternatively, or in addition, someembodiments may determine similarity using other metrics, such asproviding a count of n-grams that are identical or share a root n-gram.For example, some embodiments may determine a similarity score between acomputer-generated query and a user-provided query based on the numberof n-grams that are shared between the two queries, where the similaritymay be a sum of the number shared n-grams.

In some embodiments, the process 1600 may include determining whetherthe similarity score satisfies a set of criteria, as indicated by block1644. In some embodiments, satisfying the set of criteria may includesatisfying a similarity score threshold. For example, if the similarityscore is provided as a distance value in a sentence embedding space,some embodiments may determine that the similarity score satisfies theset of criteria if the similarity score is greater than a similarityscore threshold. A sentence embedding space may include a vector spacehaving dimensions determined from a parameter-tuning operation, wheredifferently-sized vector spaces may be used based on a corpus of textdocuments. Alternatively, or in addition, some embodiments may satisfy aset of criteria by having a minimum number or minimum ratio of sharedn-grams. For example, after determining a similarity score between acomputer-generated query and a user-provided query based on a sharednumber of n-grams, some embodiments may determine whether the similarityscore satisfies a minimum ratio of shared n-grams or a minimum number ofshared n-grams. If a determination is made that the similarity scoresatisfies the set of criteria, some embodiments may proceed tooperations described for block 1646. Otherwise, operations of theprocess 1600 may proceed to operations described for block 1648.

In some embodiments, the process 1600 may include retrieving thedocument using the map to the first document stored in the index, asindicated by block 1646. The map to the first document stored in theindex may be stored in the form of a record of a database, an array, abalanced search tree, another type of m-ary tree, a trie, or anothertype of index data structure. Some embodiments may load the index or aportion of the index based on a determination that a user is associatedwith a context value mapped to the index or the portion of the index.For example, a user associated with the domain category value“cardiology” may be provided with a first index assigned to usersassociated with the domain category value “cardiology.”

Some embodiments may use the first set of learned representations toretrieve the first document via the index. For example, some embodimentsmay, after a determination that a learned representation of auser-provided query is sufficiently similar to a learned representationof a computer-generate query, use the first learned representation as akey to access the index. Using the key may permit some embodiments toretrieve a value mapped to the key, where the value may be an identifierof the first document or otherwise mapped to the first document. Forexample, using one or more of the operations described above, someembodiments may determine a similarity score based on a differencebetween a first sentence vector “[12, 53, 0]” and a second sentencevector “[13, 53, 1].” After a determination that the similarity scoresatisfies a similarity threshold, some embodiments may use the firstsentence vector as a key to access an index in order to retrieve a linkto a first document that is mapped to by the key. Using a sentencevector or another learned representation of a query as a key may includeusing the vector or other learned representation directly or using anoutput of a function that takes the learned representation as an input.For example, some embodiments may use the elements of a sentence vectorto navigate through nodes of a search tree to reach a leaf node storingan identifier of a document and a text position of the document. Variousoperations may be performed when navigating through an index. Forexample, some embodiments may then retrieve a document via a trie bystarting at a root of the trie and recursively traversing nodes of thetrie using a key based on a first set of learned representations of thecomputer-generated query to reach a leaf node. In some embodiments, theleaf node may one or more types of values identifying a document or datain the document, such as a pointer to the document, an identifier of thefirst document, a section of text in the document, or the like.Alternatively, or in addition, some embodiments may compute a hash valueof the vector and use the hash value as a key to navigate through anindex. Alternatively, or in addition, some embodiments may use then-grams of the query directly, where the sequence of n-grams may be akey of the index that leads to a document.

In some embodiments, the process 1600 may include retrieving a documentwithout using the map of the first set of learned representations to thefirst document, as indicated by block 1648. Some embodiments mayretrieve a document based on a user-provided query by performing a setof operations described elsewhere in this disclosure. For example, someembodiments may retrieve a document based on the user-provided query byreplacing one or more n-grams of the query with an alternative set ofn-grams, where the alternative set of n-grams may be determined via aset of ontology graphs. Alternatively, or in addition, some embodimentsmay use an index to retrieve a document, where a key of the index may bedetermined based on the second set of learned representations or someother value based on the user-provided query.

In some embodiments, the process 1600 may include displaying a searchresult in a user interface based on the user query, as indicated byblock 1656. Some embodiments may display an n-gram sequence used togenerate the query that led to the document being displayed in a UIbeing displayed on a screen of a user computing device or anothercomputer system. In some embodiments, the UI element displaying then-gram sequence may show the n-gram sequence surrounded by other textthat neighbor the n-gram sequence. For example, if a first n-gramsequence is the first phrase “celery is green” and is in a firstsentence stating, “the dog does not like the fact that celery is greenand crunchy,” some embodiments may provide a UI that displays the firstsentence. The first phrase may be visually indicated and distinct fromthe surrounding text via highlighting, bolding, text font change, orsome other visual indicator. Similarly, some embodiments may displaytabular data, where rows, columns, or specific entries of the tabulardata may be visually indicated in the UI.

As described elsewhere in this disclosure, some embodiments may use aset of ontologies to update a query used to retrieve one or moredocuments. Some embodiments may provide a web message, programinstructions, or other data that causes a UI to display text from adocument directly. For example, after obtaining a query “are catsdangerous,” some embodiments may use one or more operations described inthis disclosure to generate the query “feline danger assessment” anddetermine a learned representation based on the query “feline danger.”Some embodiments may then use the learned representation as a key toretrieve a document and text positions in the document via an index thatwas generated using one or more computer-generated queries orcorresponding learned representations. The UI may then be updated todisplay text indicated by the text positions, where the text may recite“assessments of feline danger.” Some embodiments may highlight the text,where the word “feline” may be highlighted to indicate that it isassociated with the word “cats” via an ontology graph.

As described elsewhere in this disclosure, some embodiments may displaya sequence of related text sections of one or more documents. Forexample, as described above, some embodiments may retrieve a set ofdocuments related to a first document. Some embodiments may cause a UIto display a first text section in the UI and a UI element permittingthe display of other text sections in the UI. In some embodiments, thedisplay of subsequent text sections may be visually depicted in variousforms, such as cascading sheets, cards, a visualized pathway of text.For example, after obtaining a first query, some embodiments may displaya set of text sections via a set of UI elements. A first UI element mayinclude a first text summary of a first retrieved document and includean interactive component. The interactive component of the UI elementmay cause the UI to display a second text section or text summary of asecond document after an interaction with a user, where the second textsection may have been retrieved based on a user context parameter. Forexample, some embodiments may display the summarization “Malaria is adisease” in response to obtaining a query “is malaria a disease” via aUI box, where the UI box includes an interactive UI element that, uponinteraction, may display a summarization of a second section of thefirst document. Alternatively, or in addition, an interaction with theinteractive UI element may cause the display of a second document thatis retrieved based on a user context parameter, such as an indicatedpreferred domain category value. For example, the UI element may causethe display of the sequence of n-grams, “your travel history indicates arisk of contracting Malaria,” which may be obtained from a seconddocument.

FIG. 17 is a conceptual diagram of a workflow for generating orotherwise updating a query, in accordance with some embodiments of thepresent techniques. The workflow 1700 displays a set of relatedsubsystems by which a query may be generated or expanded. In someembodiments, an initial query 1704 may be obtained and include acomputer-generated query or a user-provided query. After the initialquery 1704 is provided by a user or generated by a computer system, someembodiments may perform a first set of operations represented by block1710. When performing the first set of operations represented by block1710, some embodiments may access a first ontology 1712 labeled with thedomain “tech” to replace or augment one or more n-grams of the initialquery 1704. Alternatively, or in addition, some embodiments may generatea set of additional queries based on the initial query 1704, where eachrespective additional query may use one or more n-grams mapped to by thefirst ontology 1712.

Some embodiments may then update the first query or generate a secondset of queries based on a second set of operations represented by block1720. When performing the second set of operations represented by block1720, some embodiments may access a second ontology 1722 labeled withthe domain “legal” to replace or augment one or more n-grams of theupdated query with n-grams from the second ontology 1722. Alternatively,or in addition, some embodiments may generate a set of additionalqueries based on the updated query or initial query 1704, where eachrespective additional query may use one or more n-grams mapped to by thesecond ontology 1722. Some embodiments may further update the initialquery 1704 based on n-grams indicating a shared or otherwise relatedconcept between the first ontology 1712 and the second ontology 1722.

Some embodiments may then update the first query or generate a third setof queries based on a third set of operations represented by block 1730.Performing the third set of operations may include using a transformerneural network or other neural network. For example, some embodimentsmay use a transformer neural network 1732 to translate a query into atranslated query 1734. Other embodiments may perform other operationswith other transformers, such as generating a text summary, generating aquery, or the like.

FIG. 18 is a logical architecture indicating data flow through a dataingestion system, ontology-based language system, domain datasets, andinformation retrieval system, in accordance with some embodiments of thepresent techniques. The logical architecture 1800 includes a dataingestion system 1804, where the data ingestion system 1804 may performone or more operations described in this disclosure to obtain one ormore documents of a corpus of documents. Some embodiments may also usethe data ingestion system 1804 to perform one or more operations toobtain other types of data such as image data, video data, interactivemedia data, or the like. Some embodiments may then perform one or moreoperations to augment the other types of data with associated text data,such as transcripts of audio generated from video or interactive mediadata, words recognized from an image, or the like.

After ingestion, some embodiments may provide the data to the languagesystem 1808, where the language system 1808 may include a knowledgefabric that includes the ingested data. In addition, some embodimentsmay use a data augmentation system to associate or augment the corpususing a knowledge-processing system 1820. Using the knowledge-processingsystem 1820 may include generating or updating a set of ontologies 1824based on the knowledge fabric 1812, where the set of ontologies 1824 maythen be used to indicate or update data associated with the knowledgefabric 1812. Various other operations may be performed by theknowledge-processing system 1820 to increase the speed and accuracy ofdata retrieval and analysis operations on the knowledge fabric 1812.Such operations may include determining one or more sets of embeddingvectors of documents in the knowledge fabric 1812, performing one ormore query expansions with a query expansion subsystem 1822, or thelike.

The language system 1808 may be used to provide a set of domain datasets1830. The set of domain datasets 1830 may include data from theknowledge fabric 1812 augmented with data provided by theknowledge-processing system 1820. Some embodiments may then access theset of domain datasets 1830 when using the information retrieval oranalysis system 1840. As described elsewhere in this disclosure, someembodiments may further augment the set of domain datasets 1830 with aset of indices 1832, where the set of indices 1832 may have beengenerated by the language system 1808 using one or more operationsdescribed in this disclosure. For example, the language system 1808 maygenerate a set of queries based on text from documents in the knowledgefabric 1812, where some embodiments may generate or update the set ofindices 1832 based on the set of queries. Some embodiments may furtheraugment the set of domain datasets 1830 with the domain-specific dataset augmentation system 1834 to include data specific to an application,where the application may use or modify the information retrieval oranalysis system 1840. Some embodiments may use the information retrievalor analysis system 1840 by using a search system 1842, where the searchsystem 1842 may obtain a query or provide text or other data in responseto a query. As described elsewhere in this disclosure, the provided datamay include the set of domain datasets 1830, other data stored in theknowledge fabric 1812, other data provided by the knowledge-processingsystem 1820, other data stored in the language system 1808, other dataingested by the data ingestion system 1804, or the like.

V. Ontology-Augmented Interface

A user interface (UI) allows users of varying expertise to updateontology graphs or other data described in this disclosure. A UI mayinclude UI elements that display text or other information, provide away for a user to provide inputs, reconfigure the UI, provide a meansfor a user to interact with a program in communication with the UI, orperform other operations. A text-displaying UI may include features thatincrease the efficiency of navigating and viewing information stored ina document, such as a scrollbar, a text search function, wordhighlighting, or the like. However, a UI that does not include visualindicators or otherwise detect text based on domain-specific data mayincrease the difficulty of adapting a document for viewing by differentusers. In addition, a UI that indicates n-grams mapped todomain-specific ontologies may be less comprehensible or useful fordocument comparison operations or operations to provide users with a wayto update domain-specific ontologies.

Some embodiments described in this disclosure may update a weight, bias,or other model parameter associated with an n-gram mapped to a vertex ofan ontology graph. As described elsewhere in this disclosure, an updateto a n-gram in a UI may update an association between a first n-gram andan ontology graph by generating a vertex mapped to the first n-gram,deleting the vertex, or modifying the vertex. The update to the n-grammay cause additional updates to other operations, such as updates to oneor more machine learning operations, query expansion operations,document retrieval operations, or the like. For example, as furtherdescribed below, some embodiments may update a machine learningoperation based on an update to a text document in a user interface. Byaugmenting a user interface with an updated ontology graph, someembodiments may reduce the computation time required to perform dynamic,user-specific content display in a user interface. Such time reductionsmay be especially helpful when dealing with large corpora of data, suchas corpora having more than 1000 documents, more than 100,000 documents,or more than 1,000,000 documents.

Some embodiments may accelerate or otherwise improve the efficiency ofone or more operations described in this disclosure by updatingontology-specific indices or domain-specific indices based oninteractions with a UI. As described elsewhere in this disclosure, anindex may map n-grams to other n-grams and may be indicated asaccessible to a set of user accounts or categories associated with useraccounts. By updating an index based on one or more updates caused byinteractions with a UI, some embodiments increase the accessibility andease for a domain expert to create or maintain an ontology that is thenusable to generate visual indicators of text associated with anontology.

In some embodiments, as described elsewhere in this disclosure, a UI mayprovide users with the ability to graphically update a data ingestion orprocessing workflow. For example, some embodiments may provide userswith a UI that represents a workflow as a set of workflow blocks. Theworkflow blocks may represent operations, models used during theoperations, corpora ingested during the operations, arguments usedduring the operations, or other elements of a workflow. Differentconfigurations of the workflow blocks or other UI elements may indicatean order of operations or relationships between embodiments, where auser may modify the configuration when sending instructions to update aworkflow.

FIG. 19 is a flowchart of operations to for updating a user interfacefor displaying text of a document, in accordance with some embodimentsof the present techniques. Operations of the process 1900 may begin atblock 1902. In some embodiments, the process 1900 may include obtaininga set of context parameters, as indicated by block 1902. As describedelsewhere in this disclosure, a set of context parameters may beobtained from a user account and may include a set of account parametersassociated with the respective user identified by the user account. Forexample, a user may be logged into a corresponding user account during adata session between a client computing device and a server, wheremessages sent between the client computing device and the server mayidentify a user account. The set of user account parameters may includeone or more categories indicating a domain of expertise, a domain classwithin a domain of expertise, another type of subdomain within a domain,other domain category values, or the like. Furthermore, as describedelsewhere, some embodiments may also obtain one or more contextparameters based on a query or other history of activity associated witha user. For example, based on a set of words provided by a user in ahistory of queries, some embodiments may determine that a user isassociated with a first domain and its corresponding ontology graph.

In some embodiments, the process 1900 may include retrieving a set ofontology graphs based on the set of context parameters, as indicated byblock 1904. As described elsewhere in this disclosure, the set ofcontext parameters may directly identify one or more ontology graphsavailable to a user. Alternatively, or in addition, some embodiments maydetermine a set of user roles or other user categories associated with auser and determine a set of ontology graphs based on the set of userroles or other user categories. For example, some embodiments maydetermine that a user account is labeled with the user role “Level 4specialist,” and retrieve a set of ontology graphs for use. Someembodiments may distinguish between a first and second set ofontologies, where a user may have read-level access for the first set ofontologies, and where the user may have read-write-level access to thesecond set of ontologies. For example, some embodiments may retrieve afirst and second set of ontology graphs. A user may read one or moredocuments labeled with n-grams of the first ontology graph but not bepermitted to edit the first ontology graph, whereas the same user may bepermitted to update the second set of ontology graphs by addingadditional words, phrases, or other n-grams.

In some embodiments, the process 1900 may include determining whether toupdate a UI for updating a data ingestion or processing workflow, asindicated by block 1906. Some embodiments may determine that a user isattempting to update a data ingestion processing workflow based on amessage provided by a client computing device being used by the user.For example, some embodiments may receive a web message from a clientcomputing device indicating that a user is requesting access to a UIwindow or other UI element to update a data ingestion or processingworkflow. In response, some embodiments may determine that the UI shouldbe updated to permit the modification of a data ingestion or processingworkflow. Some embodiments may first determine whether the user has anappropriate permission or user role to update a data ingestion orprocessing workflow. For example, a user having the user role “dataengineer” may be permitted to update a data ingestion or processingworkflow, whereas a user having the user role “tourist” may be preventedfrom updating a data ingestion or processing workflow. If adetermination is made that a UI should be updated to modify a dataingestion or processing workflow, operations of the process 1900 mayproceed to block 1910. Otherwise, operations of the process 1900 mayproceed to block 1928.

In some embodiments, the process 1900 may include sending a UI to modifya data ingestion and processing workflow based on the set of contextparameters, as indicated by block 1910. Operations to update the dataingestion and processing workflow may include one or more of theoperations described further below for the process 2000. For example,some embodiments may store the data ingestion and processing workflow inthe form of block text obtain a higher-level language. The dataingestion and processing workflow may be stored in various forms and maybe stored in a pre-compiled form that is then used to generate compiledmachine language code or a graphical UI. For example, some embodimentsmay receive a web message indicating that a user wishes to access afirst workflow. Some embodiments may then retrieve a datasetrepresenting a data ingestion and processing workflow. For example, someembodiments may retrieve data encoded a YAML format including squarebrackets and curly brackets.

In some embodiments, the process 1900 may include obtaining a firstmessage requesting a set of documents of corpora, as indicated by block1912. As discussed elsewhere in this disclosure, a message requesting aset of documents may be provided in the form of a query withoutidentifying a specific document, where some embodiments may send thedocument in response to the query. For example, some embodiments mayobtain a first message including a query for a document that includesthe question, “what are the side effects of aluminum?” In response, someembodiments may retrieve a plurality of documents based on the query.Alternatively, or in addition, some embodiments may receive a messagedirectly identifying the document. For example, some embodiments mayobtain a message including an identifier for a document and, inresponse, directly send the document to the client computing device sothat it may be rendered for viewing in a UI.

In some embodiments, the process 1900 may include determining data for aUI that causes the display of text from the set of documents and a setof visual indicators based on the set of ontology graphs, as indicatedby block 1914. Some embodiments may send the data in a plurality ofpackets, where a message may be distributed across a plurality ofpackets. For example, some embodiments may send data over a plurality ofrequest-response exchanges between a server and a client computingdevice. Some embodiments may provide some or all of the data using a setof third-party services, such as a content delivery network (CDN). Forexample, some embodiments may send UI data to a client computing devicevia a CDN over multiple responses that are part of a set ofrequest-response exchanges, where the data may include text data, imagedata, metadata associated with other data, or the like.

Some embodiments may send UI data that includes program code that, whenexecuted by a client computing device, causes the display of a UI, wherethe program instructions may include scripting code such as JavaScriptcode, pre-compiled program code such as web assembly code, or the like.For example, some embodiments may provide program code that causes a UIbeing displayed on a client computing device to render text from anatural-language text document, where the rendered text includes a setof visual indicators indicating one or more words or other n-grams thatmap to a set of vertices of an ontology graph. Furthermore, as describedfurther below, some embodiments may send UI data that includesstructured data interpretable by a native application that is alreadydisplaying a version of the UI, where the structured data may be used toupdate the display of the UI. For example, some embodiments may send aJSON file to a client computing device, where the client computingdevice may use a native application to interpret the JSON file andupdate a UI based on the JSON file. As described elsewhere in thisdisclosure, a visual indicator may include highlighting, text bordering,colored text, an animation, or the like.

For example, some embodiments may display a paragraph of text, where afirst word of the paragraph is highlighted in a first color to indicatethat the first word is associated with an ontology via a vertex of theontology.

Some embodiments may provide a UI that displays visual indicatorsassociated with different ontologies. For example, a section of textbeing rendered for presentation by a UI may include a first n-gram “IgG”and a second n-gram “Child.” A first visual indicator may indicate thatthe first n-gram is mapped to a vertex of a first ontology graph labeledwith the domain “medical tests.” A second visual indicator may indicatethat the second n-gram is mapped to a vertex of a second ontology graphlabeled with the domain “demographics,” where the first and secondvisual indicators may use different colors, be surrounded with differentborders, or otherwise be visually distinct from each other. For example,some embodiments may identify a vertex of a second ontology graph basedon the second n-gram by determining an embedding vector based on thesecond n-gram and then matching the embedding vector with a set ofembedding vectors mapped to vertices of the second ontology graph.Alternatively, or in addition, some embodiments may identify a vertex ofan ontology graph by determining an embedding vector of the secondn-gram, determining the closest embedding vector to the embedding vectorof the second n-gram based on a distance in an embedding space, andselect the vertex mapped to the closest embedding vector.

Alternatively, or in addition, a UI may display one or more n-grams andan associated set of visual indicators indicating that the set ofn-grams is mapped to a plurality of ontologies or subdomains within theplurality of ontologies. For example, a UI may display an n-gram and avisual indicator indicating that the n-gram is mapped to verticesassociated with different ontologies. Various configurations may bedisplayed in a UI to identify the set of ontologies associated with avisual indicator. For example, a first ontology identifier “domain 1”and a second ontology identifier “domain 2” may be displayed in a visualindicator surrounding an n-gram to indicate that the indicated n-gram ismapped to vertices of a pair of graphs identified by “domain 1” or“domain 2.”

Some embodiments may provide a UI that includes one or more UI elementsthat may be interacted with to send a set of requests to a server basedon an input or configuration of the UI. For example, the UI may causethe client computing device to send a second message to a computersystem, where the second web message may include an n-gram indicated bya user and an update value corresponding with the n-gram, where theupdate value may indicate a change to a vertex or an addition to avertex. Various types of updating operations may be performed, where then-gram may be updated to be a different n-gram, may be associated with anew ontology graph, or the like. For example, the UI may include a UIelement in the form of a button with the rendered text “submit changes.”

After an interaction with the UI element by a user, the interaction mayinclude a click with a cursor displayed on a computer monitor or tap ona touchscreen, a client computing device may provide a web messageindicating one or more user-provided update. In some embodiments, theupdate may include a request to update an ontology. An ontology updaterequest may include a request to update an n-gram mapped to theontology, remove an n-gram from an ontology, or add an n-gram to anontology. For example, an ontology update request may include a firstn-gram, a domain category value, and a function argument indicating thatthe first n-gram should be removed from the ontology graph(s)categorized with the domain category value.

Some embodiments may provide a UI that indicates or permits a user toupdate relationship types between different vertices via n-grams mappingto the different vertices. Various UI element interactions may be usedor combined to cause a client computing device to send a second messageto update an ontology graph. For example, a user may highlight the word“coughing,” drag the highlighted word to the word “tuberculosis” in theUI, and indicate that “coughing” is associated with “tuberculosis” via afirst relationship type “relType1” selected from a dropdown menu of theUI. The user may then tap on a UI element such as a button labeled“submit,” causing the client computing device to send a web message thatincludes the n-gram “coughing” and a set of update values including“tuberculosis” and “relType1” to a server performing one or moreoperations described in this disclosure. The client computing devicedisplaying the UI may then send a web message that includes thehighlighted n-gram and the set of update values, where the set of updatevalues may indicate a change to a vertex or an addition to a vertex.Some embodiments may determine that an update value indicates a changeto a vertex or an addition to a vertex based on the update valueidentifying a vertex. For example, the update value “tuberculosis” inthe message described above may indicate that the update valueidentifies the vertex based on the update value “tuberculosis”identifying a vertex of an ontology graph. For example, as discussedfurther below, some embodiments may update a vertex such that theprevious n-gram mapped to the vertex is replaced with a replacementn-gram identified by one or more of the update values.

Some embodiments may provide a UI that concurrently displays a first anda second natural-language text document. For example, some embodimentsmay provide a UI that displays a first text document that is a previousversion of a second text document and concurrently display the first andsecond text documents for a document comparison operation.Alternatively, or in addition, some embodiments may provide a UI thatpresents the second text document, where text differences or otherchanges to the second text document with respect to the first textdocument may be indicated. In addition to indicating changes, someembodiments may indicate text differences associated with a domaincategory, generate or otherwise update a visual indicator to indicatethe domain category, or notify a user that data associated with thedomain category was updated. For example, an n-gram present in a firsttext document may be absent in a second text document, where the n-grammay be mapped to a first ontology associated with a first domaincategory. Some embodiments may then determine that the updated versionchanges one or more text sections associated with the first domaincategory based on the absent n-gram. In response, some embodiments mayupdate a visual indicator to include an identifier of the domaincategory or notify a user that a text difference associated with thedomain category has occurred.

As described elsewhere in this disclosure, a user account may includeaccount parameters that indicate a set of domain category values. Insome embodiments, the set of domain category values may be organized asa sequence of domain category values or otherwise be used to establish ahierarchy of domain category values for the user account. The hierarchyof domain category values may then be used to configure an arrangementof UI elements on a display. For example, some embodiments may provide aUI that displays a set of changes between two documents, where the setof changes include changes to text including n-grams associated withdifferent domains or other categories. Some embodiments may then selectwhich set of changes to prioritize and display above other changes basedon a hierarchy of domain category values associated with a user account.For example, some embodiments may provide a UI that displays a firsttext section associated with the category “infectious disease” and asecond text section associated with the domain category “organ failure.”In response to a determination that a user account is associated with ahierarchy of domain category values prioritizing the category “organfailure” over the category “infectious disease,” some embodiments maydisplay the second text section visually associated with the first textsection on a display screen of the UI. Being visually associated withthe first text section may include being above the first text sectionand within a pixel range of the first text section, where the pixelrange may include values less than 10 pixels, values less 50 pixels,values less than 100 pixels, values less than 200 pixels, or the like.Similarly, some embodiments may display the first identifier “organfailure” in visual association with the second identifier “infectiousdisease,” such as by displaying the first identifier above the secondidentifier, where at least one character of the first identifier iswithin 100 pixels of a pixel of a character the second identifier.

Some embodiment may use a set of ontology graphs to determine if a setof expected n-grams are missing from a document. For example, someembodiments may determine whether a first set of n-grams mapped to aconcept of an ontology graph is present in a document based on anothern-gram mapped to the shared concept being present in the document as aset of alert criteria. In response to a determination that the first setof n-grams mapped to the first concept is not present in the document,some embodiments may determine that one or more alert criteria have beensatisfied. Based on a determination that an alert criterion issatisfied, some embodiments may notify a user by sending a message via amessaging platform, updating a UI to display an alert message, sending amessage to a user account, or the like. By using one or more ontologygraphs to determine which n-grams to detect for a set of alert criteria,some embodiments may increase the adaptability of a system to detectingmissing information for various documents by taking advantage of anontology graph's structure with respect to associating disparaten-grams.

Some embodiments may use a set of ontology graphs to determine whether atext section of a document is repetitive and provide a metric ofrepetitiveness when providing or updating a UI. For example, someembodiments may count the number of n-grams of a set of n-grams mappedto a set of vertices are used, where each vertex of the set of verticesis associated with a shared concept. In response to a determination thatthe count satisfies an alert threshold (e.g. by exceeding the alertthreshold) or some other alert criteria, some embodiments may notify auser. The use of a set of ontology graphs to measure repetitiveness ordetermine a set of alert criteria may reduce the memory requirements formeasuring document repetitiveness. By using an ontology graph determinedbased on a set of context parameters, some embodiments may adaptivelyselect which words and concepts to count based on a set of user accountparameters. Furthermore, by using an ontology graph, some embodimentsmay determine semantic repetitiveness with greater accuracy.

Some embodiments may indicate one or more logical contradictions basedon associations between vertices mapped to n-grams of a document. Forexample, some embodiments may store an ontology graph having a firstvertex mapped to a first n-gram “condition1.” The first vertex may beassociated with a second vertex via a graph edge, where the secondvertex may be mapped to a second n-gram “symptom1,” and where the graphedge may be associated with relationship type “cause.” Some embodimentsmay then receive an update message from a client computing device toupdate an ontology indicating that the “condition1” and “symptom1” aremutually exclusive. Some embodiments may then determine whether the pairof relationship types “cause” and “mutually exclusive” are included in afirst list of relationship type pairs indicated as being contradictory.In response to a determination that the pair of relationship types“cause” and “mutually exclusive” are included in the first list ofrelationship type pairs, some embodiments may generate an alertnotification based on a determination that the logical contradiction hasbeen detected.

In some embodiments, the process 1900 may include sending data for theUI to the client computing device, as indicated by block 1916. Asdescribed elsewhere in this disclosure, sending a UI may include sendingdata interpretable by a web browser displaying a UI, a nativeapplication that includes the UI, or the like. In some embodiments, dataassociated with the UI, such as a bytecode version of the UI orcomponents of the UI, may be sent to a computing device. For example, aweb browser or other application executing on a client computing devicemay obtain a bytecode version that includes pre-interpreted libraries orframeworks and compile the bytecode to an executable binary encoding ofthe bytecode version. Some embodiments may store one or more functionsbased on an ontology graph in a bytecode format that is then sent to aclient computing device, where the ontology graph or an index based onthe ontology graph may be sent to the client computing device.

Alternatively, or in addition, some embodiments may provide UI datawritten in a markup language such as JSON, XML, or the like that maythen be interpreted by a UI executing on a web browser, as a part of anative application, or another computing platform. For example, someembodiments may first provide a set of UI elements encoded in apre-compiled bytecode format to a client computing device that may thenbe displayed on a web browser. After an interaction with a user, someembodiments may then send a set of structured data stored as a JSONdocument to indicate one or more updates to the UI. By storing andsending UI data in a structured data format, some embodiments mayincrease the reliability and transferability of UI configurationsbetween different computing systems and users.

Some embodiments may reference the uncompiled or compiled version of theUI in a subsequent data session to reuse elements of the UI stored in acache. For example, some embodiments may perform operations to copyontology graph data or other data used by a UI from a first memoryaddress space to a second memory address space, where the second memoryaddress space may be in persistent memory. By copying data to a localpersistent memory of a client computing device, some embodiments mayreduce the network cost of rendering data stored in corpora or provide ameans of performing one or more of the operations described in thisdisclosure without requiring a connection to a server.

In some embodiments, the process 1900 may include obtaining a secondmessage to update the set of ontology graphs, as indicated by block1920. Operations to obtain the second message to update the set ofontology graphs may include operations similar to those described forblock 1912. As described elsewhere in this disclosure, the message toupdate the second set of ontology graphs may include instructions toupdate a vertex mapped to an n-gram, where the n-gram may be providedvia the second message. For example, some embodiments may obtain thesecond message in the form of a web request sent by a client computingdevice that includes a first n-gram and a set of update values includinga second n-gram and instructions to replace the first n-gram with thesecond n-gram. Some embodiments may obtain an n-gram of a message thatdirectly includes the n-gram. Alternatively, or in addition, a messagemay include an n-gram by including an n-gram identifier that isdifferent from the n-gram itself, such as a globally unique identifier(GUID) for a word. For example, some embodiments may obtain a messagethat includes the GUID “2151223x3126,” where “2151223x3126” may bemapped to the n-gram “selectively.”

In some embodiments, the process 1900 may include updating a set ofontology graphs based on the second message, as indicated by block 1924.Updating a set of ontology graphs may include adding, removing,modifying a variable of, or otherwise updating a vertex of the set ofontology graphs. Some embodiments may update a set of ontology graphs byupdating a vertex of a first graph of the set of ontology graphs tochange an associated n-gram from a first n-gram to a second n-gram. Forexample, some embodiments may, after receiving a message indicating anupdate to the n-gram “borscht” to an update value “borst,” someembodiments may select a first vertex mapped to the n-gram “borscht.” Insome embodiments, the first vertex may be directly mapped the n-gram“borscht.” Alternatively, or in addition, the first vertex may be mappedto the n-gram “borscht” via an embedding vector generated based on then-gram “borscht.” After selecting the first vertex, some embodiments maymodify its associated n-gram with the update value “borst.”Alternatively, or in addition, some embodiments may update the set ofontology graphs by updating indices generated based on the set ofontology graphs. For example, some embodiments may update a set of trienodes of an index to replace the n-gram “borscht” with “borst.”

In some embodiments, updating the set of ontology graphs may includeadding a vertex to the set of ontology graphs. For example, someembodiments may receive a message indicating that an n-gram should bepart of an ontology graph. Some embodiments may then determine whetherthe n-gram is already mapped to a vertex of an ontology graph. Based ona determination that the n-gram is not mapped to any vertices of theontology graph, some embodiments may update the ontology graph toinclude a new vertex that maps to the n-gram. In addition to adding thenew vertex to the ontology graph, as described elsewhere in thisdisclosure, some embodiments may also include associations between thenew vertex and other vertices of the ontology graph.

As discussed above, updating the set of ontology graphs may includeupdating a set of graph edges of the ontology graph associated with ann-gram. For example, some embodiments may obtain a message indicatingthat a first n-gram mapped to a first vertex of a first ontology graphis associated with a second n-gram mapped to a second vertex of a secondontology graph. The message may include a first n-gram and a set ofupdate values including the second n-gram and a relationship typebetween the first and second n-grams. In some embodiments, afterreceiving the message indicating the first n-gram is associated with thesecond n-gram, some embodiments may update a set of ontology graphs byadding an ontology graph edge that connects the first vertex with thesecond vertex. For example, some embodiments may access a list of graphedges consisting of an array of vertex identifier pairs representinggraph edges and add a new vertex identifier pair to represent anassociation between the first and second vertices. Alternatively, or inaddition, some embodiments may update an index based on the updatedontology graph.

Some embodiments may update a weight, bias, activation functionparameter, or other neural network model based on user interaction witha UI. For example, some embodiments may receive a message from a clientcomputing device based on a user interaction that indicates that theuser interaction should generate or modify a relationship type between afirst and second vertex of an ontology graph. By receiving a messageindicating instructions to generate or modifying a relationship typebased on a UI interaction between a pair of n-grams, some embodimentsmay generate or modify a graph edge or other association between thepair of vertices mapping to the pair of n-grams. Some embodiments maythen update a machine learning model by update the training of thelearning model based on the newly-generated or modified relationshiptype. For example, some embodiments may determine embedding vectors inan embedding space for an n-gram in a sequence of n-grams based on othern-grams of the sequence of n-grams. After determining that a graph edgebetween a corresponding pair of vertices mapped to the pair of n-gramshas been generated or updated, some embodiments may change an n-gramweight or other value used to determine the embedding vector. In someembodiments, an update to the value used to determine the embeddingvector may cause further updates to a set of neural network weights,biases, activation function parameters, hyperparameters, or otherlearning model parameters during a training operation of a machinelearning model. Alternatively, or in addition, some embodiments mayupdate model parameters of statistical models based on the userinteraction with the UI.

In some embodiments, updating the set of ontology graphs, machinelearning models, or other elements of program code may include compilingor recompiling program instructions. For example, as described elsewherein this disclosure, some embodiments may perform querying operationsbased on the set of ontology graphs. Some embodiments may performcompiling operations when updating the set of ontology graphs. Variouscompiling systems may be used, such as a LLVM compiling system ormulti-level intermediate representation (MLIR) compiler. For example,some embodiments may use a LLVM compiling system that compiles a sourcelanguage to an intermediate representation (IR) and optimize the IR withan LLVM IR optimizer.

As described in this disclosure, an intermediate representation mayinclude program instructions structured for further processing that isagnostic with respect to a source or target programming language. Forexample, an intermediate representation provided by the LLVM compilingsystem may provide a set of program code, where each respectiveinstruction of the program represents a fundamental operation. The IRmay be provided in one or more various forms, such as a three-addresscode, graph-based form, stack-based form, or some combination thereof.In some embodiments, program code to determine semantic relationshipsbased on an ontology may be compiled into an abstract syntax tree orother IR. Furthermore, as discussed further below, some embodiments mayuse a compiler adapted for compiling machine learning operations, suchas TVM.

In some embodiments, the process 1900 may include determining whether aset of decision trees should be updated, as indicated by block 1930. Asdescribed elsewhere in this disclosure, one or more decision trees orother decision system may be used by a user to determine what additionalactions to take, to categorize a record, or to perform other operations.In some embodiments, the decision tree may be used as a form of naturallanguage instructions, where updates to the decision tree may correspondwith updates to natural language instructions based on the conceptsassociated with the decision tree. Unless otherwise stated, it should beunderstood that operations used to update a decision tree may be used toupdate other types of decision systems, and that other decision systemsmay be used instead of or in conjunction with a decision tree to providea decision.

In some embodiments, data stored in a set of ontology graphs may affectthe decision tree. Some embodiments may determine a set of decisiontrees that are affected by an update to a set of ontology graphs andwhether the set of decision trees should be updated based on a set ofcategories indicated by the set of ontologies described above. Forexample, some embodiments may determine a set of domain category valuesbased on the vertices being updated by a message sent from a clientcomputing device, where the set of decision trees is associated with theset of domain category values. The domain category value may include anidentifier for a domain of knowledge, a class within the domain ofknowledge, an identifier of a concept or other subdomain, or the like.For example, some embodiments may update a first vertex of a firstontology graph based on a first n-gram, where the first ontology graphmay be labeled with the domain title “medicine” with a domain classvalue “expert.” After determining which domain categories have beenupdated, some embodiments may then determine a set of affected decisiontrees based on the ontology graph by selecting the set of decision treesassociated with the updated categories “medicine” and “expert.”

Some embodiments may first determine whether the updated domain islisted in a set of decision-impacting domains. In response to adetermination that the updated domain is listed in the set ofdecision-impacting domains, some embodiments may update a decision treeassociated with the updated domain. For example, some embodiments maydetermine that ontology graphs associated with the domain category“medicine” have been updated and then determine that the domain category“medicine” is listed in the set of decision-impacting domains. Inresponse, some embodiments may determine that a first decision treeshould be updated, where the first decision tree is listed inassociation with the domain category “medicine.” Alternatively, or inaddition, some embodiments may determine a set of decision tree nodes,each respective node corresponding to a respective decision tree of aset of decision trees based on the domain category value or anassociated set of vertices.

Some embodiments may accelerate the speed of decision tree updateoperations by generating and maintaining an index that directlyassociates a domain category value with a set of decision tree nodes orother elements of a decision tree. For example, some embodiments maydetermine that a vertex associated with the domain category value“symptoms” has been updated. Some embodiments may then use an index thatassociates the domain category value “symptoms” with a set of affecteddecision tree nodes to determine that one or more decision trees shouldbe updated. Alternatively, or in addition, some embodiments may use aset of values that are associated with vertices that identify a set ofdecision tree nodes. For example, some embodiments may access a firstvertex representing a concept, where the first vertex may identify a setof decision tree nodes, and where each respective decision tree nodecorresponds with a different decision tree. After an update to the firstvertex or an update to another vertex adjacent to the first vertex, someembodiments may then determine that a set of decision trees listed inassociation with the first vertex should be updated. If a determinationis made that a set of decision trees should be updated, operations ofthe process 1900 may proceed to operations described block 1938.Otherwise, operations of the process 1900 may proceed to operationsdescribed for block 1942.

In some embodiments, the process 1900 may include updating a set ofdecision trees based on the updated set of ontology graphs, as indicatedby block 1938. Updating a decision tree may include updating one or moreelements of a decision tree such as updating a set of labels (e.g., amedical diagnosis), updating a decision operation represented by adecision tree node, updating a vocabulary of n-grams used by thedecision tree, or the like. Various types of updates to a decision treemay be made. Some embodiments may update a decision tree by updating aset of n-grams used to make a labeling decision or associations betweenthe n-grams used to make the labeling decision. The set of n-grams maybe updated by replacing a previous n-gram of the set of n-grams with newn-gram, deleting an n-gram from the set of n-grams, adding an n-gramthat causes the selection of a second n-gram at a decision tree node, orthe like. For example, some embodiments may replace one or more previousn-grams in a set of n-grams used at an operation represented by adecision tree node to label a record, such as by replacing the n-gram“hypertension” with the n-gram “hypertensive crisis” in response to auser's interaction with a UI. Some embodiments may then update logiccorresponding to a decision tree node based on the update to the set ofontology values. For example, after an update to the decision tree, someembodiments using the updated decision tree may assign the diagnosticlabel “emergency” to a patient after a user selects the term“hypertensive crisis” using a medical diagnosis program.

In some embodiments, an updated decision tree may be used by an engineto label data. For example, an updated decision tree may be interpretedby a rules engine to label a medical record with a diagnosis based on anontology graph or n-grams mapped to the ontology graph. In someembodiments, a decision operation represented by a decision tree nodemay use selected n-grams of different sets of n-grams to determine alabel or another decision tree decision result. For example, someembodiments may use logic corresponding with a decision node thatdetermines whether a patient has the diagnosis “viral infection” basedon whether a first n-gram of a first set of n-grams and a second n-gramof a second set of n-grams are selected during a medical checkup usingan electronic medical record (EMR) system.

Alternatively, or in addition, some embodiments may use an updateddecision tree to change one or more system operations or mechanicaloperations. For example, some embodiments may obtain input values from aclient communication program as inputs for a decision system that usesthe updated decision tree. Some embodiments may then categorize, label,or otherwise indicate one or more values based on the decision nodes ofthe decision tree. For example, a decision may have been updated toindicate that the domain “stop request” includes the sequence of n-gram“cancel services,” where an indication of a stop request may cause anNLP system to stop a program using natural language processing programcode (e.g., a chatbot). Some embodiments may then receive a web messageindicating that an NLP system has received a client communication thatincludes the phrase “cancel services” and, in response, stop theexecution of an NLP program code being used to communicate from theclient.

In some embodiments, a user may be notified based on the update to theset of decision trees. In some embodiments, a decision tree may beassociated with a list of user accounts or a list of categoriesassociated with user accounts. The list of user accounts or categoriesassociated with user accounts may be used to notify users in response toa determination that the categories associated with user accounts may beupdated. For example, some embodiments may determine that a list ofcategories associated with an updated decision tree includes the userroles “administrator” and “doctor.” Some embodiments may then select aset of user accounts associated with the user roles and, for eachrespective user account, send a respective notification message via amessaging communication platform, an e-mail, a SMS text message, or thelike. For example, after determining that an update to an ontology graphvia a first user's interaction with a UI displaying rendered text from adocument causes an update to a decision tree, some embodiments may senda notification message to a second user indicating that the decisiontree has been updated.

In some embodiments, the process 1900 may include updating the UI basedon the updates to the set of ontology graphs, as indicated by block1942. Operations to update the UI may include one or more operationsdescribed above for block 1914. As described above, some embodiments maysend the message to update the UI to indicate that the change to then-gram described above has been performed. For example, some embodimentsmay obtain a request to update an n-gram that is associated with a firstontology graph to instead be associated with a second ontology graph.After performing one or more operations described above, someembodiments may then send a message to the client computing device thatcauses the UI to update its display of rendered text to include a visualindicator that indicates that the n-gram is mapped to the secondontology graph.

FIG. 20 is a flowchart of operations to for updating a user interfacefor updating a workflow, in accordance with some embodiments of thepresent techniques. Operations of the process 2000 may start at block2002. In some embodiments, the process 2000 may include obtaining a setof context parameters and a corresponding set of ontology identifiers,as indicated by block 2002. Operations to obtain the set of contextparameters or corresponding ontology identifiers may be similar tooperations described elsewhere in this disclosure. For example, the setof context parameters may be obtained from a user account, determinedfrom data obtained from a client computing device being used to accessthe user account, or determined from queries or other inputs provided bythe user.

In some embodiments, the process 2000 may include retrieving a dataingestion or processing workflow, as indicated by block 2004. Retrievingthe data ingestion or processing workflow may include obtaining aworkflow that is automatically loaded for a user to update.Alternatively, or in addition, some embodiments may retrieve the dataingestion or processing workflow after receiving a message to retrievethe workflow based on an identifier provided in the message.

Retrieving the data ingestion or processing workflow may includeretrieving data encoded in one or more various data serializationformats, such as a JSON, XML, YAML, or the like. For example, someembodiments may retrieve a set of data including a structured datadocument that is written using a data serialization format representedin the form a bracketed data such as, ‘{“name”: “pipe1”, “pipes”:[{“name”: “pipe0”, “sources”: [“https://1f9i3tng.xml”], “steps”:[{“type”: “transformer”, “value”: “xml_src_cr_doc”, “args”:[ ] . . . .’As by the bracketed data, different values enclosed in differentbrackets may include different elements of a workflow, such as a name ofa neural network model used process data or a name of a data ingestionsource used to add documents to corpora. Some embodiments may thendynamically generate a user interface based on the workflow data, wherea sub-element (e.g., a list within a list) in the bracketed data may beconverted into smaller shapes that are then fit into larger shapesrepresenting an element that includes the sub-element. As describedfurther below, some embodiments may then dynamically update a UI withworkflow blocks or other UI elements to represent a set of workflowoperations to ingest and process data. Some embodiments may thengenerate an updated version of the structured data document.

Alternatively, or in addition, some embodiments may obtain a specificrepresentation of a UI configuration corresponding to the workflow. Forexample, some embodiments may retrieve a set of values indicating theposition of UI elements representing one or more operations of a dataingestion or processing workflow, where the set of values may berepresented by ‘[[0, “start”, {“collapsed”:true, “xcor”:0, “ycor”: 0,“heading”:0,}3], 100, 100, [null, 53, null]], . . . .’ As describedfurther below, some elements of the UI display data may be linked toelements of the workflow data via an identifier shared between the twoelements.

In some embodiments, the process 2000 may include sending the UI to theclient computing device, as indicated by block 2008. Operations to sendthe workflow may include one or more operations similar to operations tosend or update a UI as described elsewhere in this disclosure. Forexample, some embodiments may send a web message including program codeto a client computing device. Furthermore, as described elsewhere inthis disclosure, the UI may include one or more UI elements that permita user to update data sources, model selection, model parameters, orother elements of a data ingestion and processing workflow.

in some embodiments, the process 2000 may include obtaining an update tothe data ingestion and processing workflow, as indicated by block 2012.Operations to obtaining an update to the workflow may include operationssimilar to those described for operations to obtain web messages orother messages as described elsewhere in this disclosure. Someembodiments may obtain the update to the data ingestion via a messageprovided by a client computing device. The message may include dataindicating updates to the data ingestion and processing workflow, wherethe data may include elements of program data similar to the programdata for the data ingestion or processing workflow described above. Someembodiments may receive indicators of differences between the retrievedworkflow code and the updated workflow code, where the message does notinclude the entirety of the workflow code.

In some embodiments, the process 2000 may include detecting or reportingerrors, redundancies, or other detected issues based on the update tothe data ingestion and processing workflow, as indicated by block 2016.Various operations may be performed to determine whether a workflowincludes one of a set of detected issues. Some operations may includedetermining whether the workflow satisfy a set of criteria, wheredifferent criterion of the set of criteria may correspond with differentissues. For example, some embodiments may determine whether the workflowsatisfies a first criterion to determine whether two different textprocessing models are being used, where the two different textprocessing models are indicated to be counter-productive or redundantwith respect to each other. Some embodiments may then determine whetherthe workflow satisfies a second criterion to determine whether a set oflisted document ingestion pipelines duplicate data ingestion. Someembodiments may provide a notification to indicate whether one or moreof the issue criteria is satisfied and, in response notify a user thatthe update to the workflow may create one or more detected issues. Someembodiments may further perform operations to determine whether the dataingestion pipeline updates an ontology graph that a user may not havepermission to update.

In some embodiments, the process 2000 may include updating the UI basedon the update to the data ingestion and processing workflow, asindicated by block 2020. Operations to update UI may include operationsdescribed elsewhere in this disclosure. For example, some embodimentsmay send a web message including an encoded form of the workflow.Alternatively, or in addition, some embodiments may send an updatedversion of a UI configuration that may reconfigure the appearance of theworkflow representation in the UI.

As disclosed elsewhere in this disclosure, some embodiments may generatea set of compiled program instructions to perform one or more operationsdescribed in this disclosure. Various compilers may be used to generatethe compiled program instructions, such as Glow, TVM, MLIR, or the like.Some embodiments may use a compiler stack adapted for learningoperations, such as the TVM compiler described by Chen et al. (Chen, T.,Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang,L., Hu, Y., Ceze, L. and Guestrin, C., 2018. {TVM}: An automatedend-to-end optimizing compiler for deep learning. In 13th {USENIX}Symposium on Operating Systems Design and Implementation ({OSDI} 18)(pp. 578-594)), which incorporated herein by reference. For example,some embodiments may use a deep learning-adapted compiler to combinesmall operations, static memory planning pass, and data layouttransformations. Some embodiments may combine small operations by fusingoperators to a single kernel, such as fusing injective operators (i.e.one-to-one mapping operators) with reduction operators (e.g., summationoperators). For example, some embodiments may compile programinstructions using an operator fusion operation that includes selectingan addition operator and a summation operator and fusing the twooperators into a single operator, where the fused operator does notstore intermediate results of both the addition operator and summationoperator.

In addition, some embodiments may perform other operations such asreinterpret tensor operations optimized for specific hardwareconfigurations, cooperative data fetching, tensorizing arithmeticworkflows, or the like. Some embodiments may select the use of alearning-adapted compiler based on a determination that a workflowincludes one or more learning operations. For example, in response to adetermination that an instruction to update the UI includes aninstruction to use a neural network, some embodiments may select alearning-adapted compiler to perform one or more operations.

FIG. 21 is a diagram of an example set of user interface elementsindicating ontology-linked n-grams, in accordance with some embodimentsof the present techniques. As used in this disclosure, anontology-linked n-gram may include an n-gram that maps to a vertex of anontology graph, where the mapping may be a direct mapping or be based ona learned representation of the n-gram. The set of UI elements 2100includes a UI element 2110 and a second UI element 2130, where the UIelement 2110 is shown as a first window that includes text from aretrieved natural-language text document. A UI element may include anyelement of a UI that may be viewed or interacted with by a user.Examples of UI elements may include buttons, sliders, radio dials, modalwindows, other windows, sidebars, or the like, where a UI element mayinclude other UI elements. As used in this disclosure, an interactionwith a first UI element may include an interaction with a second UIelement if the second UI element is within or otherwise connected to thefirst UI element. For example, the UI element 2110 includes the UIelement 2132, and an interaction with the UI element 2132 may also be aninteraction with the UI element 2110.

The set of visual indicators 2111-2118 may indicate different words orother ontology-linked n-grams that are mapped to a set of vertices of aset of ontology graphs. Different respective visual indicators of theset of visual indicators 2111-2118 may correspond with differentontology graphs. For example, the first visual indicator 2111, thirdvisual indicator 2113, fourth visual indicator 2114, and fifth visualindicator 2115 may be colored with a first color to indicate that theyare associated with a first ontology graph. In addition to thecoloration of the visual indicator, the visual indicator may include anidentifier of the associated ontology graph, which is displayed as“MEDICAL_TESTS.” In addition, the second visual indicator 2112 may beassociated with a second ontology graph labeled with the domain category“CASE” and may include the text “CASE” to further indicate the domaincategory. In addition, the sixth visual indicator 2116 and the eighthvisual indicator 2118 may be associated with a third ontology graphlabeled with the domain category “DEMOGRAPHICS” and may include the text“CASE” to further indicate the domain category. In addition, the seventhvisual indicator 2117 may be associated with a fourth ontology graphlabeled with the domain category “SYMPTOMS” and may include the text“SYMPTOMS” to further indicate the domain category.

Some embodiments may provide a UI that permits a user to update one ormore ontology graphs with a UI element. For example, some embodimentsmay provide a UI that permits a user to highlight the word “Serological”being displayed in the UI element 2110 and indicate that the word shouldbe added to the second ontology graph via interactions with a set of UIelements. After updating the UI to indicate that the word “Serological”should be added to an ontology, a user may interact with the UI element2132 by clicking on or tapping on the UI element 2132 to send a messagethat indicates an update to an ontology graph.

FIG. 22 is a diagram of an example set of user interface elementsindicating comparisons between different versions of a document, inaccordance with some embodiments of the present techniques. The set ofUI elements 2200 includes a change summary window 2210 and a textcomparison window 2250. The change summary window 2210 includes a firstsummary window 2212 and a second summary window 2213. Each respectivesummary window of the first and second summary windows 2212-2213summarizes both a total number of text sections and a count of textsections corresponding to ontology graph categories.

The change summary window 2210 also includes a selection menu 2220,which provides a list of domain identifiers corresponding with differentontology graphs. Each domain identifier in the list of domainidentifiers may be presented as an interactive UI element. For example,after selecting the UI element 2226, which includes the identifier“Medical Test,” a window 2228 may present text from a first documentassociated with the domain identified by the identifier “Medical Test.”While not shown in FIG. 22, some embodiments may provide a UI thatincludes other types of domain category values, such as expertise classvalues, concepts or other subdomains, or the like. The change summarywindow 2210 also includes a tag selection window 2224 presents three UIelements such as the UI element 2225, where each UI element shows anidentifier of a domain category associated with one or more updated textsections when comparing the first document with a second document. Thethree UI elements shown in the tag selection window 2224 may correspondwith expertise class values, concepts, or other subdomains of the domainselected with the selection menu 2220 and may be used to further filterthe display text in the window 2228. For example, after a user taps onthe UI element 2225 using a touchscreen or otherwise interacts with theUI element 2225, the window 2228 may present text associated with thedomain category “ababab.” As described elsewhere in this disclosure,some embodiments may determine that the text section in the window 2228is associated with the domain category “ababab” based on an associationbetween the acronym “OBP” and the category, “ababab.”

As used in this disclosure, a first and second document may be versionsof a shared document. For example, a first document may be an updatedversion of a second document, where the second document may be stored asthe first document in combination with a set of changes to the firstdocument. As described above, some embodiments may provide a UI capableof filtering the text of a document to present only portions of the textsurrounding a text section where a pair of documents differ whencomparing the pair of documents.

In some embodiments, the two versions of the document may be twoversions of a set of natural language instructions. As describedelsewhere in this disclosure, some embodiments may display the domaincategory values most relevant to a user when prioritizing detectedchanges between an updated version and a prior version of a set ofnatural language instructions. A set of natural language instructionsmay include a flow chart, a manual for operating a device or using aprogram, a regulation or other government rule, a company policy orpolicy of an organization, or a decision tree. Some embodiments maydisplay a set of topics or other domain category values in associationwith the corresponding changes between two versions of natural languageinstructions.

While the above describes showing comparisons between two versions of asame document to track changes of the document, some embodiments maytrack changes in n-grams over time based on the use of the n-gram inmultiple documents over time. As described elsewhere in this disclosure,some embodiments may update associations between different concepts orother n-grams over time based on documents authored or otherwiseobtained at different times. For example, based on a first set ofdocuments authored before a first date, some embodiments may determinethat the n-gram “vert1” is associated with the n-grams “vert2,” “vert3,”and “vert4,” where each of the n-grams may represent concepts in adocument. After obtaining a second set of documents authored after thefirst date, some embodiments may determine that the n-gram “vert1” isassociated with an n-gram “vert5” based on an updated embedding vectorcorresponding with the n-gram “vert5,” where the n-gram “vert5” mayrepresent another concept and may be encoded in a same or differentontology graph with respect to the n-gram “vert1.” Some embodiments maythen update the set of ontology graphs, such as by appending a subarrayassociating the pair of vertices to an array of subarrays, where eachsubarray may represent a graph edge of an ontology graph. Someembodiments may then update additional operations based on a discoveredassociation between a first concept and a second concept, such as byupdating text-displaying operations to display the second concept aftera user highlights the first concept. In addition, some embodiments maystore a time corresponding to when the association between the first andsecond concept was first detected. Some embodiments may then provide avisualized representation of a time-based map of the change inassociations between different concepts or other n-grams.

Some embodiments may label a paragraph, specific text section, or othertext section with a category in response to a determination that thetext section includes one or more ontology-linked n-grams associatedwith the label. For example, some embodiments may determine that then-gram “OBP” maps to a first vertex of an ontology graph and that thefirst vertex is associated with a second vertex mapped to the n-gram“ababab.” In some embodiments, the n-gram “ababab” may represent aconcept associated with multiple vertices other vertices such as avertex mapping to the n-gram “OBP.” Alternatively, or in addition, someembodiments may determine that the n-gram “OBP” is associated with acategory labeled “ababab” via an index of categories associated withn-grams.

The text comparison window 2250 displays a set of text sections from afirst document and a visual indicator 2251 indicating a text differencebetween a text section of the first document in contrast with a seconddocument. In addition, some embodiments may indicate one or more domainsaffected by the difference. For example, each respective text section ofthe text sections 2252-2254 may be presented with a set of respectivedomain category values associated with the respective passage in ascreen location adjacent or otherwise in proximity to (e.g., within 100points of a screen). For example, a user may move a cursor 2230 over thetext section 2252 to open a window 2260, where the window 2260 mayindicate a set of domain category values including a domain“pulmonology,” a domain class “basic,” and a subdomain “pneumonia.”

The set of UI elements 2200 also includes other buttons or other UIelements that may update the data being displayed in the text comparisonwindow 2250, such as the first button 2261, second button 2262, thirdbutton 2271, and fourth button 2272. For example, an interaction withthe first button 2261 may cause the presentation of a set of domaincategory values associated with each respective text section of the textsections 2252-2254. For example, a UI may be updated after a user clickson the first button 2261, where the update may cause the domain categoryvalues “pulmonology,” “basic,” and “pneumonia” to be displayed inproximity with the text section 2252. In some embodiments, aninteraction with the second button 2262 may display only text sectionsthat have been determined to have been updated. In some embodiments, aninteraction with the third button 2271 may display a list ofnatural-language text documents from a corpus of text data that includestext sections 2252-2254. In some embodiments, an interaction with thebutton fourth 2272 may display one or more decision trees that areassociated with the document being displayed. For example, after aninteraction with the fourth button 2272, the text comparison window 2250may display a decision tree having decision tree nodes associated withone or more n-grams used in the text sections 2252-2254 or other textsections of a document.

FIG. 23 is a diagram of an example user interface displaying arepresentation of a decision tree, in accordance with some embodimentsof the present techniques. Some embodiments may provide a UI thatdisplays a decision tree, where the decision tree may be used to performoperations such as recommending an action, labeling a record, or thelike. The decision tree 2300 includes a decision tree root node 2302 anda rules engine implementing the decision tree 2300 may begin at a staterepresented by the decision tree root node 2302.

The UI displaying the decision tree 2300 may permit a user to view orupdate a first list of n-grams 2310. In some embodiments, the first listof n-grams 2310 may be associated with one or more vertices of anontology, where each of the one or more vertices may be associated witha shared label. For example, the first list of n-grams 2310 may eachrecite a first set of symptoms, where each symptom may be mapped to avertex of an ontology graph that is associated with a first sharedlabel. In some embodiments, the vertices may share a label by beingassociated with another vertex representing a concept via a set of graphedges. For example, the vertices of a first set of n-grams may share thelabel “type 1 symptom” based on each of the vertices be associated witha graph edge to another vertex that map to the n-gram “type 1 symptom.”Alternatively, or in addition, some embodiments may store an index ordata table of labels, where a record of the data table may provide alist of n-grams or their corresponding vertices of an ontology graph.

An implementation of the decision tree may then provide the first listof n-grams 2310 to a set of program instructions to provide or moredecisions based on the decision tree 2300 based on whether an n-gram ofthe first list of n-grams is provided as an input to the set of programinstructions. For example, a function of the set of program instructionsmay include presenting a second user with a list of options that includeoptions corresponding with the first list of n-grams 2310. In responseto a determination that the second user did provide an n-gram of thefirst list of n-grams as an input to the set of program instructions,some embodiments may use the decision tree 2300 to categorize a recordor perform an action based on the provided n-gram.

Some embodiments implementing the decision tree 2300 may then proceed toa decision point represented by the decision tree node 2312. The logiccorresponding with the decision tree node 2312 may include determiningwhether the second user should be permitted to select n-grams thatinclude the second list of n-grams 2320 or n-grams that include thethird list of n-grams 2330. For example, an application executing on aclient computing device may obtain a dataset representing the decisiontree 2300 via an API and use a rules engine to implement the decisiontree 2300. The application may present a first UI window that provides auser with a superset of symptoms that includes the symptomscorresponding with the list of n-grams 2310. If a user of theapplication selects symptoms corresponding with a first subset ofn-grams of the list of n-grams 2310, the application may provide theuser with a second UI window that permits the user to select follow-upsymptoms corresponding with the second list of n-grams 2320. If a userof the application selects symptoms corresponding with a second subsetof n-grams of the list of n-grams 2310, the application may provide theuser with a third UI window that permits the user to select follow-upsymptoms corresponding with the third list of n-grams 2330. For example,after a user selects a first option corresponding with the n-gram“XYZ003,” the logic represented by the decision tree node 2312 may causethe application to provide a UI displaying symptoms corresponding withthe second list of n-grams 2320.

Some embodiments may perform categorization decisions based on thedecision tree 2300. For example, some embodiments may perform acategorization operation represented by the decision tree node 2332,where a user's selection of n-grams from one or more of the list ofn-grams 2310, 2320, 2330 or 2340 may be used to perform a labelingdecision. As discussed elsewhere in this disclosure, a categorizationdecision may be based on n-grams selected or detected from a pluralityof sets of n-grams. For example, a decision system that uses thedecision tree 2300 may recommend that a patient record be labeled with“gout” in response to a determination that the n-gram “pain” is detectedfrom the list of n-grams 2310 and that the n-gram “swelling” is detectedfrom the list n-grams 2320.

As disclosed above, some embodiments may permit a user to update a setof n-grams associated with ontology graphs. For example, someembodiments may detect that a user updated an ontology graph by adding anew n-gram to the first list of n-grams 2310 and, in response, updatethe logic corresponding with the decision tree node 2312 to proceed tologic corresponding with the decision tree node 2312 if the new n-gramwas selected. Alternatively, or in addition, some embodiments may updatelogic corresponding with the categorization operation represented by thedecision tree node 2332. For example, some embodiments may determinethat a new n-gram is associated with a diagnosis n-gram representing acategory based on a relationship type update provided by a user in a UIand add the new n-gram to the fourth list of n-grams 2340. Someembodiments may then update the logic corresponding with the decisiontree node 2332 to cause an application implementing the decision tree2300 to categorize a record with the diagnosis n-gram based on adetection of the new n-gram.

As described elsewhere, some embodiments may use a compiler system, suchas the LLVM compiler system to first generate an intermediaterepresentation of the implementation of the decision tree. Someembodiments may then provide the intermediate representation to a clientcomputing device for use by the client computing device. By providingthe client computing device with a pre-compiled version of the decisiontree after an ontology update, some embodiments may reduce the bandwidthrequired to execute an application implementing the rules engine. Inaddition, some embodiments may reduce the computational resourcesrequired to implement the decision tree by generating an intermediaterepresentation of the decision tree in response an update to a set ofontology graphs.

FIG. 24 is a diagram of an example set of user interface elementspermitting the updating of a set of corpus and data processing elements,in accordance with some embodiments of the present techniques. Asdiscussed elsewhere in this disclosure, some embodiments may obtain anupdate to an ontology graph based on an interaction with a UI thatincludes a set of UI elements 2400. The set of UI elements 2400 mayinclude interactive elements that allow a user to form connection linesor other connecting shapes between visualizations that representontology graphs, machine learning models, or the like.

The set of UI elements 2400 includes a workflow block 2410 that is shownto include UI elements as a set of workflow blocks 2411-2420. Theworkflow blocks may be displayed in various forms, such as rectangles,circles, ellipsoids, or other shapes. In some embodiments, the workflowblocks may be shown to be in contact with each other. For example, theworkflow block 2411 is shown to be in contact with the workflow block2410. In some embodiments, an order of a workflow may be visuallyrepresented by a direction of the workflow blocks. Some embodiments maydisplay a next workflow operation of a current workflow operation basedon a visual association between the workflow operations. Someembodiments may visually represent an order of a set of workflowoperations by the direction in which the corresponding workflow blocksrepresenting the operations appear. For example, by displaying theworkflow blocks 2411, 2413, 2415-2417, and 2419-2420 proceeding from topto bottom, some embodiments may indicate that each workflow operation ofthe set of workflow operations represented by the workflow blocks 2411,2413, 2415-2417, and 2419-2420 are performed in sequence, starting atthe workflow block 2411 and ending at the workflow block 2420. It shouldbe understood that this relationship between spatial configuration andan order of the set of workflow operations may be changed in otherembodiments.

Some embodiments may provide a UI that to indicates specific inputs,parameters for a workflow, data sources, names, or other valuesassociated with a workflow operation. The set of UI elements 2400includes multiple workflow blocks that represent specific inputs ormodels to be used during an execution of a workflow. The workflow block2412 may indicate that the workflow block 2411 has an input value “XX,”which may indicate that the workflow operation represented by theworkflow block 2411 may have the name “XX.” In addition, the workflowblock 2413 may represent a data ingestion operation, where the data maybe provide by a hyperlink or other data source address represented bythe block 2414.

Some embodiments may permit workflow blocks to indicate relationshipsbetween workflow operations. For example, a workflow block 2450 isindicated to have the title “To collection” by the workflow block 2451.Sub-elements of the workflow block 2450 include the workflow blocks2456-2458, which may represent a neural network model, input set ofdocuments, and additional argument(s), respectively. As indicated byshared name “box1” depicted in the workflow block 2458, some embodimentsmay provide a workflow operation or result(s) of a workflow operation asan argument for another workflow operation. Similarly, the workflowblock 2415 has the title “To collection” to indicate that the workflowblock 2415 represents an execution of an operation that is furtherdefined in the workflow block 2450. The inclusion of the workflow block2415 in the workflow block 2410 may indicate that the set of operationsrepresented by the workflow block 2410 includes performing operationsrepresented by the workflow block 2450. For example, the operationsrepresented by the workflow block 2410 may include data processingoperations, such as operations to perform a transformer-based learningoperation using the machine learning model represented by block 2456based on inputs of the type “document” represented by the workflow block2457.

Furthermore, As discussed elsewhere in this disclosure, some embodimentsmay determine whether one or more alert criteria will be violated. Insome embodiments, after determining that an interaction with a UI wouldupdate a hierarchical set of graphs, some embodiments may verify whetherone or more of the set of rules or other conditions would be violated.Various conditions may be applied and tested, such as a condition thatrestrict vertices of a first type from being associated with vertices ofa second type, a condition that restricts n-grams associated with afirst concept from being associated with a second concept, a conditionthat restricts vertices associated with a first class value from beingassociated with vertices having a different class value without anappropriate user authorization, or the like. For example, someembodiments may include a condition that a user logged in via a useraccount must have an appropriate permission value before being permittedto edit a connection between a first vertex representing a first conceptand a second vertex representing a second concept. In response to adetermination that a rule would be violated by a proposed connectionbetween vertices, a verification element of the UI may change text orappearance (e.g., change a color, shape, size, or the like) to indicatethat the rule would be violated by the proposed connection otherproposed update to a set of ontology graphs.

In block diagrams, illustrated components are depicted as discretefunctional blocks, but embodiments are not limited to systems in whichthe functionality described herein is organized as illustrated. Thefunctionality provided by each of the components may be provided bysoftware or hardware modules that are differently organized than ispresently depicted, for example such software or hardware may beintermingled, conjoined, replicated, broken up, distributed (e.g.,within a data center or geographically), or otherwise differentlyorganized. The functionality described herein may be provided by one ormore processors of one or more computer devices executing code stored ona tangible, non-transitory, machine readable medium. In some cases,notwithstanding use of the singular term “medium,” the instructions maybe distributed on different storage devices associated with differentcomputing devices, for instance, with each computing device having adifferent subset of the instructions, an implementation consistent withusage of the singular term “medium” herein. In some cases, third partycontent delivery networks may host some or all of the informationconveyed over networks, in which case, to the extent information (e.g.,content) is said to be supplied or otherwise provided, the informationmay provided by sending instructions to retrieve that information from acontent delivery network.

The reader should appreciate that the present application describesseveral independently useful techniques. Rather than separating thosetechniques into multiple isolated patent applications, applicants havegrouped these techniques into a single document because their relatedsubject matter lends itself to economies in the application process. Butthe distinct advantages and aspects of such techniques should not beconflated. In some cases, embodiments address all of the deficienciesnoted herein, but it should be understood that the techniques areindependently useful, and some embodiments address only a subset of suchproblems or offer other, unmentioned benefits that will be apparent tothose of skill in the art reviewing the present disclosure. Due to costsconstraints, some techniques disclosed herein may not be presentlyclaimed and may be claimed in later filings, such as continuationapplications or by amending the present claims. Similarly, due to spaceconstraints, neither the Abstract nor the Summary of the Inventionsections of the present document should be taken as containing acomprehensive listing of all such techniques or all aspects of suchtechniques.

It should be understood that the description and the drawings are notintended to limit the present techniques to the particular formdisclosed, but to the contrary, the intention is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the present techniques as defined by the appended claims.Further modifications and alternative embodiments of various aspects ofthe techniques will be apparent to those skilled in the art in view ofthis description. Accordingly, this description and the drawings are tobe construed as illustrative only and are for the purpose of teachingthose skilled in the art the general manner of carrying out the presenttechniques. It is to be understood that the forms of the presenttechniques shown and described herein are to be taken as examples ofembodiments. Elements and materials may be substituted for thoseillustrated and described herein, parts and processes may be reversed oromitted, and certain features of the present techniques may be utilizedindependently, all as would be apparent to one skilled in the art afterhaving the benefit of this description of the present techniques.Changes may be made in the elements described herein without departingfrom the spirit and scope of the present techniques as described in thefollowing claims. Headings used herein are for organizational purposesonly and are not meant to be used to limit the scope of the description.

As used throughout this application, the word “may” is used in apermissive sense (i.e., meaning having the potential to), rather thanthe mandatory sense (i.e., meaning must). The words “include”,“including”, and “includes” and the like mean including, but not limitedto. As used throughout this application, the singular forms “a,” “an,”and “the” include plural referents unless the content explicitlyindicates otherwise. Thus, for example, reference to “an element” or “aelement” includes a combination of two or more elements, notwithstandinguse of other terms and phrases for one or more elements, such as “one ormore.” The term “or” is, unless indicated otherwise, non-exclusive,i.e., encompassing both “and” and “or.” Terms describing conditionalrelationships, e.g., “in response to X, Y,” “upon X, Y,”, “if X, Y,”“when X, Y,” and the like, encompass causal relationships in which theantecedent is a necessary causal condition, the antecedent is asufficient causal condition, or the antecedent is a contributory causalcondition of the consequent, e.g., “state X occurs upon condition Yobtaining” is generic to “X occurs solely upon Y” and “X occurs upon Yand Z.” Such conditional relationships are not limited to consequencesthat instantly follow the antecedent obtaining, as some consequences maybe delayed, and in conditional statements, antecedents are connected totheir consequents, e.g., the antecedent is relevant to the likelihood ofthe consequent occurring. Statements in which a plurality of attributesor functions are mapped to a plurality of objects (e.g., one or moreprocessors performing steps A, B, C, and D) encompasses both all suchattributes or functions being mapped to all such objects and subsets ofthe attributes or functions being mapped to subsets of the attributes orfunctions (e.g., both all processors each performing steps A-D, and acase in which processor 1 performs step A, processor 2 performs step Band part of step C, and processor 3 performs part of step C and step D),unless otherwise indicated. Further, unless otherwise indicated,statements that one value or action is “based on” another condition orvalue encompass both instances in which the condition or value is thesole factor and instances in which the condition or value is one factoramong a plurality of factors. Unless otherwise indicated, statementsthat “each” instance of some collection have some property should not beread to exclude cases where some otherwise identical or similar membersof a larger collection do not have the property, i.e., each does notnecessarily mean each and every. Limitations as to sequence of recitedsteps should not be read into the claims unless explicitly specified,e.g., with explicit language like “after performing X, performing Y,” incontrast to statements that might be improperly argued to imply sequencelimitations, like “performing X on items, performing Y on the X′ editems,” used for purposes of making claims more readable rather thanspecifying sequence. Statements referring to “at least Z of A, B, andC,” and the like (e.g., “at least Z of A, B, or C”), refer to at least Zof the listed categories (A, B, and C) and do not require at least Zunits in each category. Unless specifically stated otherwise, asapparent from the discussion, it is appreciated that throughout thisspecification discussions utilizing terms such as “processing,”“computing,” “calculating,” “determining” or the like refer to actionsor processes of a specific apparatus, such as a special purpose computeror a similar special purpose electronic processing/computing device.Features described with reference to geometric constructs, like“parallel,” “perpendicular/orthogonal,” “square”, “cylindrical,” and thelike, should be construed as encompassing items that substantiallyembody the properties of the geometric construct, e.g., reference to“parallel” surfaces encompasses substantially parallel surfaces. Thepermitted range of deviation from Platonic ideals of these geometricconstructs is to be determined with reference to ranges in thespecification, and where such ranges are not stated, with reference toindustry norms in the field of use, and where such ranges are notdefined, with reference to industry norms in the field of manufacturingof the designated feature, and where such ranges are not defined,features substantially embodying a geometric construct should beconstrued to include those features within 15% of the definingattributes of that geometric construct. The terms “first”, “second”,“third,” “given” and so on, if used in the claims, are used todistinguish or otherwise identify, and not to show a sequential ornumerical limitation. As is the case in ordinary usage in the field,data structures and formats described with reference to uses salient toa human need not be presented in a human-intelligible format toconstitute the described data structure or format, e.g., text need notbe rendered or even encoded in Unicode or ASCII to constitute text;images, maps, and data-visualizations need not be displayed or decodedto constitute images, maps, and data-visualizations, respectively;speech, music, and other audio need not be emitted through a speaker ordecoded to constitute speech, music, or other audio, respectively.Computer implemented instructions, commands, and the like are notlimited to executable code and can be implemented in the form of datathat causes functionality to be invoked, e.g., in the form of argumentsof a function or API call.

In this patent, to the extent any U.S. patents, U.S. patentapplications, or other materials (e.g., articles) have been incorporatedby reference, the text of such materials is only incorporated byreference to the extent that no conflict exists between such materialand the statements and drawings set forth herein. In the event of suchconflict, the text of the present document governs, and terms in thisdocument should not be given a narrower reading in virtue of the way inwhich those terms are used in other materials incorporated by reference.

The present techniques will be better understood with reference to thefollowing enumerated embodiments:

1. A method, comprising: obtaining, with a computer system, via anetwork, a first message from a client computing device, wherein thefirst message identifies a natural-language text document; sending, withthe computer system, a user interface (UI) to the client computingdevice via a response to the first message, the UI comprising: a displayof the natural-language text document that comprises n-grams associatedwith a set of ontologies; a visual indicator indicating anontology-linked n-gram of the natural-language text document is mappedto an entry of an ontology; a UI element that, when interacted with,causes the client computing device to send a second message to thecomputer system, the second message comprising the ontology-linkedn-gram; obtaining, with the computer system, the second message from theclient computing device after an interaction with the UI to update theontology; updating, with the computer system, the entry mapped to theontology-linked n-gram; and updating, with the computer system, adecision tree based on the update to the ontology.2. The method of embodiment 1 wherein the natural-language text documentis a first version of the natural-language text document, the methodfurther comprising: obtaining a previous version of the natural-languagetext document; determining a text difference between the first versionof the natural-language text document and the previous version of thenatural-language text document, wherein the text difference comprisesthe ontology-linked n-gram; determining a domain associated with thetext difference based on the ontology; and sending a third message tothe client computing device to visually indicate the domain in the UI.3. The method of embodiment 2, wherein the visual indicator is a firstvisual indicator, further comprising visually indicating the textdifference using a second visual indicator, wherein the second visualindicator surrounds the ontology-linked n-gram.4. The method of any one of embodiments 1 to 3, wherein the visualindicator is a first visual indicator, and wherein the entry is a firstvertex of the ontology, the method further comprising: obtaining a thirdmessage; updating the ontology based on the third message by adding asecond vertex to the ontology, wherein the second vertex maps to asecond ontology-linked n-gram; and updating the UI by adding a secondvisual indicator indicating the second ontology-linked n-gram of thenatural-language text document.5. The method of any one of embodiments 1 to 4, wherein updating thedecision tree comprises: determining whether a domain associated withthe ontology is listed in a set of decision-impacting domains; inresponse to a determination that the domain is listed in the set ofdecision-impacting domains, determining a set of decision tree nodesassociated with the domain; and send a third message indicating the setof decision tree nodes to the client computing device.6. The method of any one of embodiments 1 to 5, further comprising:obtaining a third message indicating a request to access a set ofworkflow operations associated with a structured data document, whereinsending the UI comprises: sending a set of values causing the UI todisplay a set of shapes representing the set of workflow operations,wherein: a first shape of the set of shapes corresponds with a dataingestion operation of the set of workflow operations; a second shape ofthe set of shapes corresponds with a data ingestion source, wherein thesecond shape is connected with the first shape on the UI; a third shapeof the set of shapes corresponds with a data processing operation of theset of workflow operations, wherein a parameter of the data processingoperation comprises an identifier of a neural network model; a fourthshape of the set of shapes is connected with both the first shape andthe third shape; and the set of ontologies is displayed in aninteractive UI element as an input or output for at least one shape ofthe set of shapes; obtaining an update message to the set of workflowoperations based on a reconfiguration of the set of shapes; andre-compiling program code corresponding with the structured datadocument to generate an intermediate representation of the program codein response to receiving the update message.7. The method of embodiment 6, further comprising generating an updatedversion of the structured data document after re-compiling the programcode, wherein the updated structured data document comprises: a firstset of values enclosed in a first pair of brackets, the first set ofvalues comprising an identifier of a data ingestion source; and a secondset of values enclosed in a second pair of brackets, the second set ofvalues comprising an identifier of the neural network model.8. A tangible, non-transitory, machine-readable medium storinginstructions that, when executed by a computer system, effectuateoperations comprising: obtaining, with a computer system, a firstmessage provided by a user from a client computing device indicating anatural-language text document during a session; selecting, with thecomputer system, an ontology graph based on an account parameter of auser account identifying the user; sending, with the computer system, auser interface (UI) to the client computing device via a response to thefirst message, the UI comprising: a display of the natural-language textdocument; a visual indicator indicating an n-gram that maps to a vertexof the ontology graph; a UI element that, when interacted with, causesthe client computing device to send a second message to the computersystem, the second message comprising the n-gram and an update valueassociated with the n-gram, the update value indicating an change to thevertex or an addition to the vertex; obtaining, with the computersystem, the second message from the client computing device after aninteraction with the UI element to update the ontology graph based onthe n-gram; updating, with the computer system, the vertex based on theupdate value; determining, with the computer system, a categoryassociated with the vertex based on the ontology graph; determining,with the computer system, a set of affected decision trees based on thevertex of the ontology graph; updating, with the computer system, adecision tree of the set of affected decision trees based on the updatevalue to the ontology graph; and labeling, with the computer system, arecord based on a selection corresponding with the update value usingthe decision tree.9. The medium of embodiment 8, wherein the n-gram is a first n-gram, andwherein the vertex is a first vertex, and wherein the ontology graph isa first ontology graph, the operations further comprising: obtaining athird message comprising a second n-gram and an identifier of a secondontology graph; and updating the second ontology graph to comprise asecond vertex, wherein the second n-gram maps to the second vertex.10. The medium of any one of embodiments 8 to 9, wherein updating thesecond ontology graph comprises: determining an embedding vector basedon the second n-gram; determining whether the embedding vectoridentifies any vertices of the second ontology graph; and adding thesecond vertex to the second ontology graph based on a determination thatthe embedding vector does not identify any vertices of the secondontology graph.11. The medium of any one of embodiments 8 to 10, wherein the visualindicator comprises an identifier of the ontology graph.12. The medium of any one of embodiments 8 to 11, wherein the n-gram isa first n-gram, wherein the natural-language text document is a firstnatural-language text document, the operations further comprising:obtaining a second natural-language text document; determining a set ofn-grams that are present in the first natural-language text document andnot present in the second natural-language text document, wherein theset of n-grams comprises the first n-gram; selecting a sequence ofn-grams of the first natural-language text document based on the set ofn-grams; and sending a third message to the client computing device thatcauses the UI to display text including the sequence of n-grams and thecategory associated with the ontology graph.13. The medium of any one of embodiments 8 to 12, wherein the categoryis a first category, the operations further comprising: determining ahierarchy of domain category values based on the user account, whereinthe hierarchy of categories comprises the first category and a secondcategory; and causing the UI to display an identifier of the firstcategory to be visual associated with an identifier of the secondcategory.14. The medium of any one of embodiments 8 to 13, the operations furthercomprising: selecting a set of user accounts based on the decision treeby accessing an index; and sending a notification message to eachrespective user account of the set of user accounts.15. The medium of any one of embodiments 8 to 14, wherein the vertex isa first vertex, and wherein the n-gram is a first n-gram, and whereinupdating the decision tree comprises: determining whether the firstn-gram is associated with a second n-gram via a first graph edgeassociating the first vertex with a second vertex, wherein: the secondn-gram maps to the second vertex; and the first graph edge is associatedwith a first relationship type of a plurality of relationship types; andselecting a decision tree node of the decision tree based on the secondn-gram in response to a determination that the first n-gram isassociated with the second n-gram via the first relationship type.16. The medium of embodiment 15, wherein updating the decision treefurther comprises replacing a previous n-gram of the decision tree nodewith the update value.17. The medium of any one of embodiments 15 to 16, wherein the ontologygraph is a first ontology graph, and wherein the second vertex is avertex of a second ontology graph.18. The medium of any one of embodiments 8 to 17, wherein selecting theontology graph based on the account parameter comprises: determining auser role associated with the user; determining a set of domain categoryvalues based on the user role, wherein the set of domain category valuesindicate a domain of expertise; and selecting the ontology graph basedon the set of domain category values.19. The medium of any one of embodiments 8 to 18, the operations furthercomprising: obtaining a third message comprising a second n-gram and anidentifier of a second ontology graph; determining whether the secondn-gram maps to a vertex of the second ontology graph; and in response toa determination that the second n-gram does not map to the vertex of thesecond ontology graph, update the second ontology graph to comprise avertex mapping to the second n-gram.20. The medium of any one of embodiments 8 to 19, further comprising:obtaining a third message indicating a request to access a set ofworkflow operations associated with a structured data document, whereinthe structured data document is stored in a data serialization format,and wherein sending the UI comprises: sending a set of values causingthe UI to display a set of shapes representing the structured datadocument, wherein: a first shape of the set of shapes corresponds with adata ingestion operation; a second shape of the set of shapescorresponds with a data ingestion source, wherein the second shape isconnected with the first shape on the UI; a third shape of the set ofshapes corresponds with a data processing operation; a fourth shape,wherein the fourth shape is connected with both the first shape and thethird shape; and an identifier of the ontology graph is displayed in aninteractive UI element as an input or output for at least one shape ofthe set of shapes; obtaining an update message comprising instructionsto update the structured data document based on a reconfiguration of theset of shapes; determining whether the update message indicates anupdate to a neural network model; update a set of program instructionsbased on a determination that the neural network model is updated;compiling the set of program instructions using an operator fusionoperation, wherein the operator fusion operation comprises: determiningthat a first operator is an injective operator; determining that asecond operator is a reduction operator; and fusing the first and secondoperator to form a fused operator.21. The medium of embodiment 20, wherein the n-gram is a first n-gram,and wherein the vertex is a first vertex, the operations furthercomprising: determining a count of occurrences of a set of n-grams inthe natural-language text document, wherein the set of n-grams comprisesthe first n-gram, and wherein the set of n-grams are mapped to a sharedconcept based on the ontology graph; determining whether the countsatisfies an alert criteria; and based on a determination that the countsatisfies the alert criteria, sending an alert to an address stored inthe user account.22. The medium of any one of embodiments 8 to 21, further comprisingsteps for sending the UI to the client computing device.23. A computer-implemented method, comprising: obtaining, with acomputer system, a set of natural-language text documents that discuss atopic, the set of documents containing different states of knowledgeabout the topic at different times; selecting, with the computer system,an ontology from among a plurality of ontologies that correspond todifferent domains of knowledge, the selection being based on theontology corresponding to a domain of knowledge including the topic;identifying, with the computer system, using the ontology, conceptsdiscussed in the documents; detecting, with the computer system, changesin at least some of the concepts over time based on differences betweendiscussion of the concepts in documents authored at different times;updating, with the computer system, natural language instructions on thetopic based on the detected changes in the concepts; and storing, withthe computer system, the updated natural language instructions inmemory.24. The method of embodiment 23, wherein updating the natural languageinstructions comprises: causing a user interface to be displayed, theuser interface indicating both the updated natural language instructionsand changes in the natural language instructions relative to a priorversion of the natural language instructions.25. The method of embodiment 24, wherein: the user interface includeslinks to, or displays at least some text from, documents among the setof natural-language text documents indicative of corresponding changesto the instructions.26. The method of any one of embodiments 24 to 25, wherein: the naturallanguage instructions comprise a flow chart, a decision tree, a manual,a regulation, or a policy.27. A non-transitory, computer-readable media storing instructions that,when executed by one or more processors, effectuate operationscomprising those of any of embodiments 1 to 26.28. A system comprising: one or more processors; and memory storinginstructions that, when executed by the processors, cause the processorsto effectuate operations comprising those of any of embodiments 1 to 26.

What is claimed is:
 1. A method, comprising: obtaining, with a computersystem, via a network, a first message from a client computing device,wherein the first message identifies a natural-language text document;sending, with the computer system, a user interface (UI) to the clientcomputing device via a response to the first message, the UI comprising:a display of the natural-language text document that comprises n-gramsassociated with a set of ontologies; a visual indicator indicating anontology-linked n-gram of the natural-language text document is mappedto an entry of an ontology; a UI element that, when interacted with,causes the client computing device to send a second message to thecomputer system, the second message comprising the ontology-linkedn-gram; obtaining, with the computer system, the second message from theclient computing device after an interaction with the UI to update theontology; updating, with the computer system, the entry mapped to theontology-linked n-gram; and updating, with the computer system, adecision tree based on the update to the ontology.
 2. The method ofclaim 1, wherein the natural-language text document is a first versionof the natural-language text document, the method further comprising:obtaining a previous version of the natural-language text document;determining a text difference between the first version of thenatural-language text document and the previous version of thenatural-language text document, wherein the text difference comprisesthe ontology-linked n-gram; determining a domain associated with thetext difference based on the ontology; and sending a third message tothe client computing device to visually indicate the domain in the UI.3. The method of claim 2, wherein the visual indicator is a first visualindicator, further comprising visually indicating the text differenceusing a second visual indicator, wherein the second visual indicatorsurrounds the ontology-linked n-gram.
 4. The method of claim 1, whereinthe visual indicator is a first visual indicator, and wherein the entryis a first vertex of the ontology, the method further comprising:obtaining a third message; updating the ontology based on the thirdmessage by adding a second vertex to the ontology, wherein the secondvertex maps to a second ontology-linked n-gram; and updating the UI byadding a second visual indicator indicating the second ontology-linkedn-gram of the natural-language text document.
 5. The method of claim 1,wherein updating the decision tree comprises: determining whether adomain associated with the ontology is listed in a set ofdecision-impacting domains; in response to a determination that thedomain is listed in the set of decision-impacting domains, determining aset of decision tree nodes associated with the domain; and send a thirdmessage indicating the set of decision tree nodes to the clientcomputing device.
 6. The method of claim 1, further comprising:obtaining a third message indicating a request to access a set ofworkflow operations associated with a structured data document, whereinsending the UI comprises: sending a set of values causing the UI todisplay a set of shapes representing the set of workflow operations,wherein: a first shape of the set of shapes corresponds with a dataingestion operation of the set of workflow operations; a second shape ofthe set of shapes corresponds with a data ingestion source, wherein thesecond shape is connected with the first shape on the UI; a third shapeof the set of shapes corresponds with a data processing operation of theset of workflow operations, wherein a parameter of the data processingoperation comprises an identifier of a neural network model; a fourthshape of the set of shapes is connected with both the first shape andthe third shape; and the set of ontologies is displayed in aninteractive UI element as an input or output for at least one shape ofthe set of shapes; obtaining an update message to the set of workflowoperations based on a reconfiguration of the set of shapes; andre-compiling program code corresponding with the structured datadocument to generate an intermediate representation of the program codein response to receiving the update message.
 7. The method of claim 6,further comprising generating an updated version of the structured datadocument after re-compiling the program code, wherein the updatedstructured data document comprises: a first set of values enclosed in afirst pair of brackets, the first set of values comprising an identifierof a data ingestion source; and a second set of values enclosed in asecond pair of brackets, the second set of values comprising anidentifier of the neural network model.
 8. A tangible, non-transitory,machine-readable medium storing instructions that, when executed by acomputer system, effectuate operations comprising: obtaining, with acomputer system, a first message provided by a user from a clientcomputing device indicating a natural-language text document during asession; selecting, with the computer system, an ontology graph based onan account parameter of a user account identifying the user; sending,with the computer system, a user interface (UI) to the client computingdevice via a response to the first message, the UI comprising: a displayof the natural-language text document; a visual indicator indicating ann-gram that maps to a vertex of the ontology graph; a UI element that,when interacted with, causes the client computing device to send asecond message to the computer system, the second message comprising then-gram and an update value associated with the n-gram, the update valueindicating an change to the vertex or an addition to the vertex;obtaining, with the computer system, the second message from the clientcomputing device after an interaction with the UI element to update theontology graph based on the n-gram; updating, with the computer system,the vertex based on the update value; determining, with the computersystem, a category associated with the vertex based on the ontologygraph; determining, with the computer system, a set of affected decisiontrees based on the vertex of the ontology graph; updating, with thecomputer system, a decision tree of the set of affected decision treesbased on the update value to the ontology graph; and labeling, with thecomputer system, a record based on a selection corresponding with theupdate value using the decision tree.
 9. The medium of claim 8, whereinthe n-gram is a first n-gram, and wherein the vertex is a first vertex,and wherein the ontology graph is a first ontology graph, the operationsfurther comprising: obtaining a third message comprising a second n-gramand an identifier of a second ontology graph; and updating the secondontology graph to comprise a second vertex, wherein the second n-grammaps to the second vertex.
 10. The medium of claim 9, wherein updatingthe second ontology graph comprises: determining an embedding vectorbased on the second n-gram; determining whether the embedding vectoridentifies any vertices of the second ontology graph; and adding thesecond vertex to the second ontology graph based on a determination thatthe embedding vector does not identify any vertices of the secondontology graph.
 11. The medium of claim 8, wherein the visual indicatorcomprises an identifier of the ontology graph.
 12. The medium of claim8, wherein the n-gram is a first n-gram, wherein the natural-languagetext document is a first natural-language text document, the operationsfurther comprising: obtaining a second natural-language text document;determining a set of n-grams that are present in the firstnatural-language text document and not present in the secondnatural-language text document, wherein the set of n-grams comprises thefirst n-gram; selecting a sequence of n-grams of the firstnatural-language text document based on the set of n-grams; and sendinga third message to the client computing device that causes the UI todisplay text including the sequence of n-grams and the categoryassociated with the ontology graph.
 13. The medium of claim 8, whereinthe category is a first category, the operations further comprising:determining a hierarchy of domain category values based on the useraccount, wherein the hierarchy of categories comprises the firstcategory and a second category; and causing the UI to display anidentifier of the first category to be visual associated with anidentifier of the second category.
 14. The medium of claim 8, theoperations further comprising: selecting a set of user accounts based onthe decision tree by accessing an index; and sending a notificationmessage to each respective user account of the set of user accounts. 15.The medium of claim 8, wherein the vertex is a first vertex, and whereinthe n-gram is a first n-gram, and wherein updating the decision treecomprises: determining whether the first n-gram is associated with asecond n-gram via a first graph edge associating the first vertex with asecond vertex, wherein: the second n-gram maps to the second vertex; andthe first graph edge is associated with a first relationship type of aplurality of relationship types; and selecting a decision tree node ofthe decision tree based on the second n-gram in response to adetermination that the first n-gram is associated with the second n-gramvia the first relationship type.
 16. The medium of claim 15, whereinupdating the decision tree further comprises replacing a previous n-gramof the decision tree node with the update value.
 17. The medium of claim15, wherein the ontology graph is a first ontology graph, and whereinthe second vertex is a vertex of a second ontology graph.
 18. The mediumof claim 8, wherein selecting the ontology graph based on the accountparameter comprises: determining a user role associated with the user;determining a set of domain category values based on the user role,wherein the set of domain category values indicate a domain ofexpertise; and selecting the ontology graph based on the set of domaincategory values.
 19. The medium of claim 8, the operations furthercomprising: obtaining a third message comprising a second n-gram and anidentifier of a second ontology graph; determining whether the secondn-gram maps to a vertex of the second ontology graph; and in response toa determination that the second n-gram does not map to the vertex of thesecond ontology graph, update the second ontology graph to comprise avertex mapping to the second n-gram.
 20. The medium of claim 8, furthercomprising: obtaining a third message indicating a request to access aset of workflow operations associated with a structured data document,wherein the structured data document is stored in a data serializationformat, and wherein sending the UI comprises: sending a set of valuescausing the UI to display a set of shapes representing the structureddata document, wherein: a first shape of the set of shapes correspondswith a data ingestion operation; a second shape of the set of shapescorresponds with a data ingestion source, wherein the second shape isconnected with the first shape on the UI; a third shape of the set ofshapes corresponds with a data processing operation; a fourth shape,wherein the fourth shape is connected with both the first shape and thethird shape; and an identifier of the ontology graph is displayed in aninteractive UI element as an input or output for at least one shape ofthe set of shapes; obtaining an update message comprising instructionsto update the structured data document based on a reconfiguration of theset of shapes; determining whether the update message indicates anupdate to a neural network model; update a set of program instructionsbased on a determination that the neural network model is updated;compiling the set of program instructions using an operator fusionoperation, wherein the operator fusion operation comprises: determiningthat a first operator is an injective operator; determining that asecond operator is a reduction operator; and fusing the first and secondoperator to form a fused operator.
 21. The medium of claim 20, whereinthe n-gram is a first n-gram, and wherein the vertex is a first vertex,the operations further comprising: determining a count of occurrences ofa set of n-grams in the natural-language text document, wherein the setof n-grams comprises the first n-gram, and wherein the set of n-gramsare mapped to a shared concept based on the ontology graph; determiningwhether the count satisfies an alert criteria; and based on adetermination that the count satisfies the alert criteria, sending analert to an address stored in the user account.
 22. The medium of claim8, further comprising steps for sending the UI to the client computingdevice.
 23. A computer-implemented method, comprising: obtaining, with acomputer system, a set of natural-language text documents that discuss atopic, the set of documents containing different states of knowledgeabout the topic at different times; selecting, with the computer system,an ontology from among a plurality of ontologies that correspond todifferent domains of knowledge, the selection being based on theontology corresponding to a domain of knowledge including the topic;identifying, with the computer system, using the ontology, conceptsdiscussed in the documents; detecting, with the computer system, changesin at least some of the concepts over time based on differences betweendiscussion of the concepts in documents authored at different times;updating, with the computer system, natural language instructions on thetopic based on the detected changes in the concepts; and storing, withthe computer system, the updated natural language instructions inmemory.
 24. The method of claim 23, wherein updating the naturallanguage instructions comprises: causing a user interface to bedisplayed, the user interface indicating both the updated naturallanguage instructions and changes in the natural language instructionsrelative to a prior version of the natural language instructions. 25.The method of claim 24, wherein: the user interface includes links to,or displays at least some text from, documents among the set ofnatural-language text documents indicative of corresponding changes tothe instructions.
 26. The method of claim 24, wherein: the naturallanguage instructions comprise a flow chart, a decision tree, a manual,a regulation, or a policy.